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ABSTRACT « ^ . • . . /\ v 

» In order to determine how vocational education 
^pro^rams 'should be evaluated, it is fitst necessary to define 
vocational education, to describe how vocational education programs 
are now being evaluated, to find out how x>ther federal edupation ^ 
programs are evaluated, and to suggest what Criteria might Reasonably 
be set out in future federal legislation for. the evaluation of ^ 
vocational education. Vocational education can be defined as 
encqmpasffing some 28,0Q0 institutions providing three types of 
occupational preparation, ,in nine general types of program areas, at 
four educational levels*, to meet the needs of at least seven special . 
tieeds groups. Studies by several researcher and Congressional 
hearings have demonstrated that vocational education i& being 
e\raluated on a variety^ of criteria, , with a confusing mix of data. \ 
Federal regulations, howeXrer, call'f^ each sta'te vocational 
eduction program to >e evaluated once *e very five years iii terms of 
plan^^lg and operational processes, student achievpmelnt,. student' 
employment success, and results of additional services provided to- 
special populations. Title I (Education of ^ Di^ad^^antaged Children) 
and Public Law 94-142 (Education of the Handicapped), have somewhat 
different evaluation reqiiirements than .the Vocational Education..*' 
Amendments set forth, raising^ quesHons of whetHer evaluation should 
be seen as a management and accounting activity for the federal ^ \ 
government, or as a diagnostic and planning tool for^local 'Schoo}.s 
and districts. It is suggested that the federal role in vocational 
education evaluation requirements be lessened, inasmuch as the 
federal government provides less than 10 percent of its suj^port. ft 
is also proposed that evaluation, criteria jbe developed pertaining to 
j^rogram improvement and that evaluation of labor, .market demands be 
left to th€j federal government. (KC) - ^ 
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I. Introduction 

^ ' ^ik, ' • 

How should vocatioiTal education programs be evaluated? This is the 

question addressed ih this paper. Specifically, ,addr^>ss the question , 

^from the perspective of federal voc^tion^l education legislation. In 

rder' to suggest •answers to the question*, we first address the follow- . 




ng questions, which will provide a ba^is for our answers; 

1 \ ^ • 
What is vocational education? - * -\ - * ^ 

How are vocational education programs now being evaluated? 

How are other federal education programs^ evaluated? ^ ^ 

■ ' * 

What criteria might; reasonably be set out in future federal legis- 
lation for the evaluation of vocational' education? 

Before embarking on at least brief answers t(|*these questions, this intro- 

u ' * * 

jj ^ * • 

diction describes the background to this paper. 

\ In the- hearings prior to the 1976. Education Amendments by the U.S. 

Congress, there was much criticism of past evaluation and planning efforts 

connected with vocational Education. , As/th^ Interim R eport of the NIE 

Vp^atioA^al Education Study described it: ' . 

.\. . in 1976 > Cpjpigressj heard Repeated criticisms of the 
planning procjeSs^i ^^and in particular of coordination among 
partie^j^.,involV)?df ^nd of use of current data^ The 1976 ^ " ^ 
xAmendmprfs consequently' incorpora a number of provisions 
"^ACt^^^m^ ^Q^^^^^^^ planning and evaluation procedures. * 

V^f ' 'i. ' (National Institute, of Education, 1980, p. ^9: 

The In^Tii^^^'Report go^^'^on to point out that^ one of the mechanisms by 

which Congresjfe sought to .i)^rove the degree to which planning was, informed 

by relevant data was by establishing new evAuation requirements'. Specif- 

cally Section 112(b)(1) of the .197*6 amendments stipulated that: ' 

(A)''each State, shall duping the ^ive-year period' of ^ • 
the State plan, evaluate the effectivenessW each pro*- 
- gram within the State being assisted with mnds . available 
' unSg: this Act; and the results of thes:e miuations shall 
. < be used to revise the Staters programs-^ , \. 
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(B) each State^shall evaluate, , . each such pro- ^ 
* gram within the State which purports to impart entry 
level job "skills according to the^extent to which program 
completers and leavers-- ' ' » . 

(i) find employment^ in occupations related to f 
their Straining, and 

(ii) are^ considered by their employers to be . ^ 
•well-trained and prepared for employment, \ ^ ^ ^ ' * 

^ Two themes are .appatent in the^language of this section^ First is 

\ ^ ' ^ - \ 

that evaluations of ^effectiveness shdula jbe used to revise and improve 
vocational prog^rams. Second ijs the notion that programs* which^seek to 
prepare students directly for employment should b^^ evaluated in terms of 
whether they subsequently dp find employment related to their Jrainipg, 
and are considered well-trained by their ^employers . The latter idea was 
a reflection, of a broader theme in the 1976 legislation, namely that, 
vocational .education programs should be more closely articulated with 
labor marl^t. conditions and demands for sailed labor. After >11, it is 
of little ben^t to anyone for a program to produce individuals skilled 
in; say, repair of television and radio vacuum tubes, when most such 
' instruments now use transistors* 

But if the motivations behind the 1976 vocational education evaluatio 
reqjiirements 'are easy to undersXand, it is far less clear exactly wjiat the 

* effects of the^e requirements have^en. Available evidence on this point 

^will be Reviewed in section but Vpr'^the r>0!nent suffice it to explain 

• that p^nding^re^thorization of federal vocational education in 1982, the 
Vdca^icml Education^udy was ma'ndated to analyze i^the means of assessing 
program qualit}^ and effc^tivertess" (National Institute of Education, 1980*, Append 



In connection v 



i,th th-is mahdate,\he study staff asked us to consider 
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what criteria, might reasonably be specified in federal legislation as 
bases for -evaluatiiig vocational education programs. This paper ^is one 

/ \ '^^^ 

product of tha^ consideratJi>|ij 

II. VJhat Is/Vocational Education? ^ ' . , . . 

— ' ' — r \ ; ' \ , 

* Thi^question seems a reasonable place, to be^in. The firsts^oint to 
note is that the very phrase "vocational education" connotes two different 




typ^s of goals.. The adjective '^vocational" indicates clearly that we are 
fiot referring simply to general education, but to education for vocations. 
Vocation refers, of course, not just to* work or labor-, >-btrt-to particular 
' occupational filings. Indeetfp the oldest meaning of the word in the 

- ^ ^ \' • - ^ : . 

English language reffeis to "action on the part of God in. calling a person 

to exercise some/f unction" (Oxford English Dictionary) . At the same time, 

however, the subject of the phrase is education "the process of nour- 

ishing or rearing a child or young person" (Ibid,)^* • The point. of this < 

\distinjction i^ mb.re than merely semantic, fot it is easy to forget that; 

, o ' ': ; ; " . * . . 

vocational educi^t^on is not synonymous with vocational training, that 

the" former refers to a particular form of education. 

^ P ^ - ^ . / . ^ ^ 

But before attempting . to draw out any ijnplicatiOns from this distinction, 

let me a^dr|||*'the question .of this section at a' more practical level. What 

^ exactly arelMie sorts of programs funded under federal vocational educational 

legislatiofll' The vocational education, 'enterprise is "a congery of different 

Vsyst^s aTM|not*a single system; it is decentralizeA and diversified. Its 

** ' ' ]^^« " " ' I ' 

prOgr"anBo|||nstruction are o^fertd in differed kinSs \9f institutions" The 



* Acco)rdiT#'^to' the OED, vocation rs a Slightly older word in the English 
languaf ^"''^ v \^_. 
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number of students enrojled in 'its <iide variety of programs and courses 
was 17!3 iriill^on^in 1979. (National Institute of Education, 1980), According' 

tb Bottoms (1^80>, '4he executive director of the 'American Vocational Associa- 



tion,* "Essentially vocational education provides learning experiences that 
further the occupational awareness, exploration, preparation, and special- 
ized needs of its cliepts,. typically aged 14-6Ps*' ^ ; , 

According to the"" latest report of the^Natiohal Center for Educational 
Statistics (NCES) on The Condition of Vocational Education (1980), therd 
are nearly 28,000 different institut ions na tionwide offering vocational . 
education programs. Although the NCES has described. these institutions in 
terms' of some 18 different types, it notes that **even the identification 

and descripjtion of the institutions has required in some instances the use 

i " ' .• • " * 

of general tenns that mask mamy significant even fundamental differences, 

{ ^ . \ 

\ ' ^ 
in institutionalMntent or character'' (p.. 550), The largest single type 

o^ institution pxoviMing vocational education with* federal' support is public 
• I • ' / • r . : 

comprehensive and vocational high schools , accounting for nearly 1*6,000 

scjhools . . V!^, 

1 For this level, namely the secondary level, Vocational educators "typir. 

caily describe three major purposes for the enterprise: 

I 1. In-depth' exploration of . occupations to assist in 
the career planning process • . 

2. . DeveloJ)ment of occupational competencies designed 
to be recognized for advanced placement in V>ost- 
seco*ndary programs, and 



* The NCES report ^ndiciates that at least thrjee types of institution offer- 
ing vocationa^ "'programs do ftot rcfceive 'federal vocational education funds j * 
iiamejy privaja^^condary schools, private noncollegiate postsecondary 
Si:hobls.,. andce<^espoBdence schools, - ✓ ' . 



3. Development of occupational doinpeJ:eficies necessary to 
enter an occupation, \ J 

(Minnesota Department of Edupt;ion, 1978, p. 361) 

Vocational education programs focusing on the latter purpose some^ 

mes are described as occupational vocat;Lpnal education programs, whereas 

rograms serving mainly the f orme^ 'two purposes are called non-occupational 

1 ' ' . ^ 

in the sense that they are itat aimed at directly preparingvindividuals for 
occupations. * , ; ^ - - 

Substantial nunbers of secondary vocational stu^f^s continue beyond 
higlij school, for some^form of postsecondary schooling. -FoTTow-up data, on the 
high sch<5ol class of 1972 indicate that at some time within four, yeajs out 
of high s6hool 54%'of vocationaT high school graduates ^roll in st)me 'f orm 
of =postsecondary education programs (including both academic^and vocational 
postsecondary programs^, as compared with 62.5% of general higb school** 
gradtual^es (Woods S Haney, 1981, Appendix D) . This trend towatd pursuit 6f 
postsecondary' schooling by secondary vocational graduates appears to have 
been recognized implicitly by Congress in the 1976 Education Amendments in 
Stipulating tha.t "in no case can pursuit of. additional education or training 
by program completers' or leavers be cansidered negatively in these evalua- 
tdons" of (Occupational' vocational programs "(Section 112(b)(1)). Thus, 
foll^iw-up studies of secondary vocational graduates usually do (and certainly 
should) distinguish between' individuals who are in the labor force and those 
•who are out of the labor force--^the latter referring to individuals who are 

enrolled in school full-time, are homemakers,, and several other categories"' ~ 

* - » * 

of individuals as well.- - ' ; • 

This. description only begins convey some of ,tKe complexity of the 
vocational education enterprise, it is also worth noting that v,ocational . 



program offerings are traditionally described in terms of nine program 

areas : * • . , * 

' Agriculture ^ * , 

Distributive ^ ^ ' - 

Health _ ^ . 

Consumer and homemaking . ^ . ' ^ ^ 

Occupational home economics 
Industrial arts ^ • • ^ 

.Office occupations / *• ' ^/'^^ 

Technical , . ' ^ ^ 

Trade and industrial 

Neverthefless , when these types are disaggrejgatedf only for occupation- 

r ^ ' ^ ^ ^ *\ 

ally specific programs, instructional programs keyed to nearly 100 different* 

^occupations can be identified ( Conditi^on of Vocational .Educatidn , 1980, 

pp. 583-586), and these 100 occupational ly specific instructional programs • 

are reportedly offered at all four levels of. the vocational education enter-- 

prise (secondary, postsecondary adult-short-term, and adult-long-term). 

Such diversity^ is 'all the more impressive when it is noted that enrollment3 

in -occupational ly specific vocational education programs make up onLy about 

7 million of the estimated 17 millidn enrolments in vocational education 

programs' overall in 1978-79 (including both occupational and nonoccupational 

programs) , # 

One final aspect of diversity in the votatiojial enterprise deserves to 

^ • : ■ ■ ' ■•' 

be mentioned, namely, >the population of individuals int^ndjedi to be served* 

As far bacl^ as the Smith-Hughes Act in 1917, the -target populat.ion for 

federally aided vocational education programs has generally been understood* 

... 1 • ^ 

to be high-school aged youth preparing for occupations.* iioWever, the ^ " 



* Wootfe 5 Haney, 1979, pr6vides a more, detailed account of the evolution 
q,f federal legislative goals- in this regard. 



Vocational Education Act of 1963 greatly elaborated the definition of the* 
pbpulation to be seWed. by vocational education* Specifically, the 1963 
Act established the intent for vocational -education to meet the nee^s of 
four groups: (l)./youth in- high school;. (2) youth with special ne^s; 
(3) youth 'and adults ^in full-time postsecondary programs preparing to enter ^ 
the labor market; and (4) youth and adults unemploy/sd ar at work, and 
needing training or^retraining to^achieve employment stability. The 
special needs group was defined in the 1963 Act to 'include "persons^ who * • 

: ' , . 

have academic, socioeconomic, or other handicaps that prevent them from 

succeeding in the regular, vocational education program/' While 'the 

'•physically handicapp^ed'l were'^xplicitly excluded from the special needs 

• « 

group identified in 1963, it was added .as a special needs target group for 
vocational education by the Education Amendments of 1968, Moreover, the 
Amendments^ of 1976 further extended the special needs definition to include 
women,- limited English-speaking persons, and American Indians, 

In 'sum, the vocational education enterprise is tremendously cfomplex 
and variegated. It encompasses, at last coiint, some ;28, 000 institutions 
providing vocational education instruction. It $pans three quite different 
types of occupational preparation (occupational exploration^^ preparation 
for advanced placement in subsequent occupational preparation programs, and 
preparation for. dirett , entry into occupations). Focusing only on occupa- 
tiorial vocational education, we see nine different general' types of program 
areas (spannfng neajily 100 different occupations for which vocational pro- 
grams seek- to prepare individuals). Moreover, vocational education programs 
exist at four different levels of the educational system (secondary, 



post secondary, adult short-terl 



and adult long-term). Finally,' and. on top 



of all this, we see that vocatiorfal e'ducation also is intended to meet 
^ the needs of at least seven specifically identified special needs groups. 

How Are Vocational Educatjion Programs Now Being Evaluated? ^ 
Given an "enterprise of thi^ fnagnitudfe and diversij;^, how are vocifional 
education programs* now being evaluated? Before answering fhis question, 
^* -.let me describe at least brief 1^ what is meant by evaJuation\ In its root 
sens&>X£L^evaluate means to determine the value ot worth of. a thing. Pro- 
gram ev.aluation as a major so^cial enterprise in the United States has ' * 

/ • I 

developed only sihce 1965, when the Elementary and Secondary Education Act • 

was passed* by the .U.S. Congress with explicit provisions for the evaluation • 

of prpgrams funded under the Act. Many different sorts of activities have 

taken place in the name of evaluation since then, including needs assess- 

ments, cost effectiveness studies, planned variation experiments , program 

documentation, and Studies modeled after literary criticism. Nevertheless, 

for the purpose of this paper we adopt the/ definition of evaluation used in 

a recent Cpngresslonally mandated study of evaluation, of federally funded 

elementary- and secondary education programs, as follows: ^ • 

Evaluation is defined*. • . as. a study designed to ^ 
answer questions about wjiat a program does in the 
intere^St of making judgments about the program. The . • 
' questions often "^addressed include: Who is served? ^ • 

What services are delivered? At what cost? With 
what effect? / , j - - 

i(Boruch S Cordray, 1980, pp. 1-4)* 

* See'Chapter 2 6f the Boruch S Cordray repott for a brief review of 
some oj the diverse meanings attaehied to the word evaluation by 
different sorts of, ojbservers, * • ^. 
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• Given tl)is definition, how are such questions being answ«^e^ with 

respect to vocational education? 'I^erhaps the gasiest place to begia • 

answering this question is with a description of intent; A fuj.1 description 

of the evaluationVequirements for vocational education as described in 'the 

Boruch 6 Cordray (1980) study is provided in Appendix 1 of this paper. * 

^ I ' ' ^ . % ' t. 

Here, let me note simply 'that while such requirements have been set out 

'i . * , . ' ^ * ' 

at levels of both<» federal and state government, and contain clear specif^.- 

cations of organizations and .officials to be involved in evaluating voca- 
tional education, the purpose of ^this paper ,is not to consQ^cter such pro- 
cedural-* matters , but instead to. analyze what criteria might r-easonab'sV, be • 

• • ♦ 

set out for evaluating vocational education programs at^he local level, 

or to be more precise- vocational' education programs .receiving federal funds. 

Federal regulations currently call for each State Department of Edu- ^ 
cation to evaluate- thfe effectiveness of each such vocational eclucation 
progrttra, once within the fiv^-ye'fer period of the state plan in .terms of ; ' - 

(1) planning and operational processes;/ 

(2) student achievefnent ; • * ^ 

' t6) student employment success, and 

V (4) results of additional services provide^ to special 
i population?. 

The e^jaluation requirements specified in federal regulations in October 
1977 were considerably Inore extensive than the criteria mentioned in the 
Educatio^ Amendments of 1976. Thus, before answering the question of ^ \ 
how /rocational education programs ate now being eval^iated, let us describe at 
fea^t briefly how the criteria mentioned in the 1976 legislation were tyans-r 
formed into those set out in .the 1977 regulations,- 



12 



As notedjin the introduction^ in hearings reading to the passage oj^ 

the 1976 Amendments, there was considerable criticism of past evaluation 

and planning with respect to vocational education. Congress was particularly 

concerned that vocational education program offerings were not responsive 

to labor market demands and to manpower needs. Thus Congress specified that 

occupational vocational education programs should be evaluated in terms of 

which program completers and leavers: 

i. find employment in occupations related to their training, ^* 
(and) 

ii, ar? considered by their employers to be welKtrained and 

prepared for employment, • ^ 
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As Hendrickson (1981) recounts, vocational educators '^objected strenu- 
ously" to these^valuation 'requirements. First it was argued that such 
employment criteria do not acknowledge the broader goals of vocational * 
education programs, and the fact that vocational education xt^s more than ■ 
' simply job training. Second, it was maintained that such job placement and 
performance criteria in 'effect would hold vocational educators accountable 
to unfair standards, . ' * " 

School systems should be' held accountable for imparting 
certain knowledge and skills and for making certain that 
there is a good fit between what is taught and the jobs 
available. However, what jobs students actually take is , 
determined by a host of economic and personal factors 
_ beyond the school 's , control , *Hold us accountable for 
employabjlity , but not employment,' is the way hiany vocar 
tional educators phrased the concern, 

(Hendrickson, 1981, p, 7) 
k' Third, some people^ voiced the concern that if vocational education pro- 
grams were held accountable in terms of job placement and employer satis-- 
faction criteria, that there might be a tendency to avoid accepting indi^ 

4 

viduals* into vo^^^ional programs who might be harder to place, ''such as 
members of minority groups, the disadvantaged, or women ^ in non traditional 

i 13 ' • 



pro^grains—:in short, tKe very, groups Congress most wanted t^tfcational 

education) to reach \(Hendrickson, 1981, p. 7), 

\ * * 7 * *! ' , ' ' * * , 

§ Apparently at least partly as a result of such objection?, regulations 

issued- by the U.S. Office of Education in October 1977 specified a broader 
approach to ^evaluating vocational education/ As meutioned already, the 
regul^ations. specified' four are^s iYi, which evaluation should be conducted 
(namely, planning and qperational proceslses, student achievement, employment 
success, and results of ^additional services provided to special legi^la- 
tions>. Details on the exact criteria suggested in the regulations pnder 
each of these rubrics are presented in Table 1. > . • 

The- evaluation regulations were further elaborated .a policy memo- 
randum in^April 1979. According to Hendrickson (I98iyii^he evaluation 

- • * it" 

policy memo was issued only after **more than a year and o^ne-half of intpmai 

struggle an$ debate within OE over how prescriptive to be" setting out 

evaluation requirements. On one hand, th^re, wa^ a desire to allow states 

'and localities 'flexibility to^organize and cajr-ry out evaluations suitable 
^'^ 

to local conditions and hence to -provide more leeway for such evaluations 
to yield locally relevant infoymatign suitajrle fot program improv^ent (and, 
of course, at the same time to ayoid' oVer-regulatipn) , On the other hand, 
however, there was a*desire/to be sufficiently prescriptive so that evalua- 
tions from different 'localities yould contain comparable* data which* could 
be aggregated t^o pr^ide I ftationarl picture of the status of vocational ' 

education. / • 

The 1979 rdgulations 'apparently were, something of a compromise between 
these two. tendencies, though Hendricksoh (1981) Suggests that ''the speCi-^ 
fications in' the memoVatidvim flow more from a cortcern ... to provide a 
national picture than jfrom a concern for program improvement" (p^ 12)., 
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' ] Table 1 : ' , ^ 

, Evaluation Specifications in the Regulations 

The State board shall, 4pri^g five-year period of the Sbate ^ 
plan, evaluate in quantitative terps the ^effectiveness of each formally ' 
organized program or project supported by Federal, State, and local funds,* 
These evaluations shal^l'be in terms of; 

(a) Planning and operational processes, such as; 

Cl) Quality and availability ot instructional offerings; j- 

» (2) Guidance, counseling^' and'placement and follow-up 

V services; * • / * ' ' , ' 

, 1 ^ (3) Capacity and condilfion af facilities ^^^n^L-equipment; 

(4) Employer participation in"" cooperative programs of 

, vocational* -education; . ' . 

(5) Teacher/pupil ratios; and . , • 

(6) Teacher qualifications. 

Xb) Results of student achievfem^ent as measured, for example, by: 

(1) Standard occupational proficiency measures; 1 
o (2) Criterionr referenced tests; and , ' \ 

(3) Other examinations ,of students* skills, knowledge, 
" * attitudes, and readinfess for entering employment , 
successfully. , \ ^ 

(c) Results of student emplojiahent success as Measured, for example,- 
by: 



\ 



(1) ' **«Rat'es of employment and ^uneraployment; . 

(2) Wage rates; ' ^ 

(3) Dur^tl^on of ^employment; and v 

' (4)' Employed' satjLsfactibn with performance of vocational 

• education students as compared with performance, of persons 
who have not had vocational education.' ' * ' ^ 

(<i) The results of additional services, as measured by the suggested 
criteria under paragraphs (2), (b), and (cX of this .section, •that 
the State-provides^- under the Act to these special populations;" 



'(1) Women; " ^ ^ 

, ■ C2). Members of minority groups; ^ 

^ (3) Handi'capped persons; * ^ 

< (4) Disadvafttag^ed persons; and r 

(5) Persons of limited English-speaking ability,' . ? 

/ • • 

Source;- Federal Register , Vol,^ 42, No. IQl, October 3, 1977, Sec. 104,402. 



Whatever the balance of motivations behind it, the 1979' memo set out 



' four general guidelines: 



(1) It defined the terrti ''program'' which was the eriWty to 
* be evaluatea, 

(2) It allowed sampling of programs within states and 
: institutions for purposes of evaluation* 

(3) It encouraged cyclical patterns of evaluation so that 
programs of par,ticular types could be evaluated only 
once within, the five«-year cycle of each state's 

five-year plan, and ' * ^ 

• <^ 

(4) It set out the policy that'OE would not aggrepte into 

. national data summaries . eith^er program review .data or* 
student achievement results, though it would review and 
sujnmari2.e such results to show national trends. 

Before describing evidence on how these evaluation requirements appear 

to be working out in. practice, we ^hotfld .explain one aspect of the national 

effort to evaluate vocational education! In addition to setting out 

9 . ' / \ 

eval-uation requirements for states and program^% the 1976 Amendments alsd 
directed the ^^ OE to develop a national system for reporting vocational 
education data* This system, , subsequently known as VEDS (Vocational Educa- 
tion Data System) was to be based on info'rmatdon derived from mandated 
evaluations of vocational- education, as well, as, other information* The 
main idea behind VEDS was that it would provide a sys.tem for acquiring 
standardized and therefore comparable data from*.jii^fferent st^ates.and locali/ 
ties which^ could be aggregated so as to produce' a national picture of the 

' • / ■ . 

vocational education entferprise* The reason for mentioning VEDS here is 
» / * * 

that reaction to the 19/76 evaluation requirements and associated regulations 

appear to have been influenced* strongly by their implementation in con- 

/ 'I 
nection with VEDS/''na]{ieIy the ef f ort ,to standartiz^key aspects of evalua- 

tion and data reportjlng across the vocational educatioji enterprise. 
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So what then has happened as. a result? Four' sources of evidence are 
available with which to add'sess this question: 

•CD *A January 1979 ^report\ the State> of the Art of Vocational 
Education Evaluation; State Evaluation Procedures anci ' ^ 
Practices by CRC. Education -and Human Devftlnpinpnt ^ 
Corporation (Smith and Holt/ 1979). ^ * 

(2)' Testimony in September 1980 Congressional Hearings on 

Current Issues ip Vocation Education (U.S. Congress, ,19^0) 
including the NCES report. The Condition of Vocational Edu> . , 

cation, whi^h represents the ^irst public report of data ^ 
gathered through VEDS, and includes a review of the quality 
of data descrifiing vocational education (in U.S. Congress, 
1980, pp. 543-861). 

C3) A December 1980 report. Implementation of the Education . * 
Amendments of 1976; A Study of State and Local Compliance ' 
and Evaluation Practi ces > by Abt Associates fBerke et al . . 
1980), and . [ 

(4) A June 1981. Survey of State Evaluation Practices >in 
Vocationkl Education (Wentling, 1981). — 

In the following paragraphs, we briefly summarize the miin findings of these 

studies conceiving criteria for evaluating vocational education j^rogranis. 

The CRC Study, January 1980; This report was based on interviews, both in • 

person and via the telephonelwith SEA staff members identified by State 

Directors of VocationallEducation as knowledgeable about evaluation pro*^ 

cedures in their stptes. .Interviews were conducted , with individuals from 

all fifty states,* The report is said to be based on "information that was 

^^T^dlable in spring 1978" (p. 2). Findings from the interviews were reported 

m four major categories: ^i.purposes of evaluation, and potential uses^ 

of findings; (2) evaluation practjices; (3) evaluation procedures; and 

(4) problems in evaluation. * ' . • 

The CRC study repotted that "many states could neither describe nor 

document how information generated by evkluations Are actually used" (pp. 4-5), 

but nevertheless ^indicated that: major purposes idlbtified by individual 



interviewed in the fifty states were as .follows (with the numbers of 
states identifying each purpose shown in parentheses) : 

• Program improvement at the state, local, oV institution level 

- Fiscal accountability to state and federal-authorities (11); 
* » • 

- Program planning at state, local or particularly^ institutionsl 
level (10); • • ' 

- Compliance with federal or state requirements to evaluate or 
report on program operations (9)j and 

- Education accountability (6), 

It is somewhat difficult to know precisely what to make of these findings • 

because the CRC repoft does not make clear exactly how some of the five 
categories of purpose (e.g. fiscal accountability, educatijjn accountability, 
and compliance) differ. Nevertheless, it seems clear from the CRC findings 

•♦•Vial- fKA inannT r*nnininn riii'mnQp rited fO"^ ^fn/^n+'^ riT-in 1 »/4i»/*o -t" ^ riT^ n 1 ijof'^ •/%t%q 

was pijpgram 'iTnprove^lent.• 
• ' ' > \ . 

CRC considered evaluation practice in terms of how states were prepar^g 
to meet the 1976 legislative mandate to evaluate (1) program quality (as 
indicated by planning and operational processes); and (2) program effective- 
ness (as indicated by student achievement and employment success, specif i- ^ 
cally whether former vocational students obt,ain ♦^jemployment in an arta 
related to their vocational training" and whethe^"empliyers are sat/isfied 

/ * 

with the performance of former vocational students'*) States were divided 
into four major categories with regard^ to their preparedness to conduct^ 
evaluations in each of three areas (program review, student achievement; 
andtfolloK-i]qp surveys) as of spring ^978: 



■15- 



-states fanning to use existing systems without revisi^qvr 



\ 



5\,-s.tates planXjng to revise existing, systems in light of the new lavf; 
S, \ ". ' 

\ Vstates developing or adopting conqjletely new, evaluation systems; 

\ States which had aislayed decision 'on how they will' comply with the 
neW\law.^ * ' 

Data ''-|)i:esented in'^the CRC report indicate th*at only two spates 

(North Caroliil^ and Utah)* were planning to>.use existing e^valuation systfems 

I 

^ 4 ^ ^ 

in each of the three areas of evaluation* Twenty-four states were in- 
*-dicated to be developing a new evaluation with respect to program review, 
student achievement, and/or follow-up surveys and for fourteen other states 

it was indicated that no information was available as of spriHg 1978 re- 

• , j * 

garding what was planned in one or 'more of the three. areas of fevaluatjion* ^ 

Of the three areas concerning which states were surveyed, studetjt .achi^ve- 

l 

ment appeared to be the one for which it was most unclear what states would 
do with respect to evaluation. 



Other sTalient findings by CRC regarding evaluation practices weire as 



.follows: 




-"Forty- two states evaluate secondary programs, thirty-seven s':ates 
evaluate post- secondary programs,- while only seventeen states eval- 
,uate adult programs^' (p, 14);* 

-''Approximately^ 28 SEA-'s Tiave not developed standards of progrkm 
'quality' or 'effectiveness'" (p. 16); 

-"Among the state standardjs vhich have been developed, most fccus on 
program inputs and processes" (p. 14); 

« 

-"Only a few>SEA's are currently using student achievement as an in- 
. dicator of program effectiveness," partly because of a lack of avail- 
able procedures for determitiing in a consistent fashion what 
..^ levels are"^ (p. 23) . 



achievement 



*These numbers .do not sum to 50, at least partly as a result of somje 
apparently inexplicable* gaps .in the CRC data. 
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Under the 'rubric of evaluation procedures the CRC study inquired into 

the state .mechanism's for evaluation in the areas of program review, student 

achievement of corapentencie^s at the end of program, and student employment 
success. * ''Program review, procedures are the most fully developed* evaluation 
procedures in vocational education" (p. 24). Among procedures 'found to be -J 
used 'for program peview were self-study, on-site visit by an outside state- 
ageijcy selected team, and feedback mechanisms to verify that recomme.ndations 
.for program improvement have been implemented <p. 24). "Proce^j^es for the 
-measurement of student achivement constitute the least developed component 



of state evaluation systems. Only one-tifth of the states currently have 
procedures for the assessment of student achievement; those j^rocedures are 
mostly in developufehtal or pilot testing stages'' (p. 31). "Student employ- 
ment success serves as a majot criteria for assessing program effectiveness. 
... Approximaterly 37 states currently have operational systems which collect 



student post-pjjogram employment data. However, only 25 percent of th$. states 
reported that they conduct employer surveys although so required ;by law" 
(pp. 31-32). 

In the realm of problems in evalua\^n the CRC report noted that "many 

administrators identified difficulties* expe\ienced either in implem'entiiig 

f 
( 

evaluations or in planning them for the future^ear," even though "some 
^state administrators appear to be reluctant to iP^cuss th^ir' state's out- 
standing probems in evaluation with individuals ^o^^de^tl^ (p. 35)^. 
Th^^ specific probems identified in the\CRC report are fallowing; 

-Insufficient staff ''and funds to conduct or monitor ^(gluations pre? 
scribe^ b>^\the legislation. 

^^-Insufficient staff, who are trained in evaluation and datS analysis, 
to design evaluation procedures and make optimal use of evaluation 
findings, once they are obtained. 
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-Lack-or— iiisufficient access "^o- [data processing] facilities to ^ 

handle; store, ^nd retrieve information in a timely fashion. ^ 

* 

-Lack of clarity abput the intent of the law; what information . * 
should be included in the evaluation to- comply with the new mandate, 

-Inaccuracy in xreporting, resulting in an incorrect assessment of 
program operations (p. 35). 
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Congressional Hearings, September IgtO " '^•/••jft,^ 

A second sburce of information on the current stattfs of vocaf^onar 
education evaluation is testimony offered in hearings hel^ before the^ House 
Committee on Education and Labor in Sept^ber 1980 (U.S. Congress, 19B0)'. ' v 
While virtually all^of the testimony is' at least indirectly ^elevan^ to * 
:he issu^ of what criteria jriight be appropriate for evaluatirig vocational • 
education programs, certain specific observations are directly pertinent . ' 
to thB current status of vocational education evaluation, and hence it Is 




such observations that we will briefly describe. 

Rolf iVulfsberg of ^he NCES offered testimony concerning the current 
condition of vocational education and the experience of t^he NCES in devel- 
oping V£DS. In describing changes in enrollment between 1977-78 and 1978-79, 
Wulf^berrg .noted that ''There were 14 states and territories that . sjiqw^d . . . 
'significant deviations frcjn^the reporjting last year, , . . enrollment-^ 
cTian'ges of over 15 percpnt in* either direction." In such cases NCES inquired 
into possible causes of. such changes and it was found that "in &bout hal^ . ' 
the cases . ^ . the state attributes | the jnpst of the change to the reporting 



of accurate data for what they .feel is the first time in certain areas. They 

attribute this to new , definitions and the requirement for student .unit 

records contained to the VEDS*' (sic, p. 10). Later in response to a 

question, Wulfsberg offered the following explanation of 'the problemVbf - 

•duplicated counts of vocational education enrollments: - ^' 

i . . * 

When NCES began the development of VEDS in 1977, many, 
factors were preventing .the collection of accurate, ^ ' ^ 

comparable, undupldc^ted enrollments across States; ^ ^ ' 

0 Some-^States mad.e'hb attempt to report unduplicated counts*; 
0 Some States reported enrollments in courses, while othet .c 
States reported program enrollments onl'y; ' • • 
^ o Some^St^tes li^eported enrollments below the 6th grade', 

while others/ reported only on students in 12th grade pro- ^' /' 
' grams which ended in specific ^sequence of courses; and * 
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Thus 



Some States only reported enrollments in ^programs * 
which were directly receiving Federal ^funds', while 
others .reported all enyollrtents in siib-baccal aureate 
vocational programs. 
duplicated counts represented only one of many factors- 



-\^blocking the achievement of our objective;^ 

In order to obtain a rough -idea of the extent to which 
these problems (particularly duplication) fctill exist, we 
compared the number of occupational ly specific vocational 
enroH^nts in grades 11-12 to the pount of all enrollments 
(vocational or nonvocation^l) in grades 11-12 on a State by ^ 
State basis. The r^ults indicate that 20 States/Outlying > 
Areas ^reported ^ccup^ipnally specific vocational enrollments 
exceeding 50 percent ^ the total- iVhile in, some cases the 
high percentage was in part d,ue to 'students below grade 11 
being incorrectly reported as in grades ll-12, duplication 
and tiie cauntxng of course enrollments (rather than program 
enrol ImentO clearly remain a proble^m. ^Because the causes 
described earlier are all present in varying degrees, we 
j:annot- specify how much of th'S^overcbunt is due to duplication 
"*in particular. As the States continue to implement student 
unit record systems, however, duplication will become less 
and less of a problem in the next two or three years, (p^ 238) 

i . ' * 

Wulfsberg went on to explain the steps that NCES has undertaken in order to 
obtain accurate unduplicate'cj- counts of enrollments in" vocational education 
programs : 

The first step Was<ithe definition of rules to be' used in 
assigning a vocational student to one and only one program. 
* ^^s.....^ The only exception! was Consiuner and 'Homemaking and Induitrial 
Arts, where a studj^nt might be counted twice — once in Zof^ 
sumer and" Homemakikg or Industrial* Arts and once in\nother 
vocational program. 

* The second step was the separation of occupational ly specific 

enroLljnents from other vgcation^l enrollments for reporting 
^ purposes-. Occupational ly specific enrollments specifically 

s * exclude, among^ others^ enrollments" in^Consumer and Homemaking 
and Industrial Arts which may include duplicated count>. Also 
excluded are emrpllments itJtelow grade 11. 'Since thei States have 
■ varying policies| concem^Xthe reporting of students in grades 
10 and -below, tl|is make's occScationally specific enrollments 
much- more-comparable on a St«e by State* basis than aity voca- 
tional counts previously available,. ' 

From the initiil development of VEDS, NCES has str|ssed the' 
need for the^ establishment of student uftit record systems which, 
by 'tReir nature, produce unduplicated,. auditable enrollment data, 
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In regulations publi3hed on May 9, 1980,, NCES listed 
auditable enrollment data as one criterion for full 
compliance with VEDS reporting requirements (104 CFR 
.117-119) • NCES^has stressed in all negotiations with 
States the need to establish unit record systems, and ^ 
most States are committed to having such systems opera-r. 
tional by the 1980-81 Veporting cycle. All States should 
have auditable data by 1981*82, according to^heir remedia- 
tion plans, (pp. 238-239). 



Mr. \/ulfsberg alsp commented on the difficulty of obtaining^^curate follow- 
up data oji the employment success of former vocational education sjtudents: 



I would like to tell you more about what is happening 
to the placement side of the picture but I must say that * 
we do not feel that the data on followup that « have been - 
provided you in the ^ast are vali^, TTxey only include a 
followup program of completers^ and have some leavers-- 
those who are known to *haVe found related employment. 

This biases the' picture and lea^jes out a very important 
part of this population, namely the leavers. It implies 
by default that program -leavers are losses to the program, 
and xertainly such 'is not the case, (p, 15) 



Several Witnesses in the September d980 hearings nevertheless testified 

: c 

that VEDS was helping to remedy past problems. Dr. Gene Bottoms, exedutiye 

director of the American Vocational 'Xssociatioji, testified that: 

While 'a variety of technical and specific, as well 
as policy.leveft problems remain, it is clear that VEDS 
is a significant improvement over past ^systems, Recogni- 
tion of and action to implement more reliable local data 
* collection practices is still needed, however, (p, 202) 

Robert Taylor, executive directoi^f the National Center for Research 
in Vocational Education, also, offered testimony suggesting that evaluation , 
^ studies of vocational education prior to the establishment of VEDS tended ^ 
to be less rigorous (p* 259). / 

Nevertheless, the VEDS system did not escape criticism. Bottoms, for 
example, criticized the costs of meeting VEDS requirements and the likely 
utility of VEDS data for state and local purposes, as follows: ^ 
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^ . The, cost of fully iiiplementing the ... VEDS m'ay " • " 
. equal the total increase , in federal appropriations fo^ 
, vocational .ducdtiof since 1976. This requirement to 
• -"--^ federal data 'n^ has placed massiv^Srcb^ * ' 

responsibilities, olstates and local school districts 

""."^'"i^'/'. ^^^^ "ill Provide much timely 

. dlJ^ptit^t''? ) ^"'^ 1-al^planning and pro^^am 

The record of the SepJember 1980 'hitrings also contains a letter from 
a com«,unitVcollege chance lor .in California, criticizing VEDS as follows: 

at 1^1 nfJ"^ °f ?T P^^^i<="larly when aggregati^d 

at the national lefel, is very questionable. Such 
# aggregation tends |o minimize or ignore significant 

regiona st^te a|d institutional differences . The • 
thf h";" ^" coJlection of such data with * ' 

the 'detail Mereni to the N.C.E.S. design is particfTlaly • 




evident in/respec 
-section. / 1 am co 
duce unbiased, va 
Of at least eq 
Financial impact 
mentation has bee 
fifty (50) millio 
even adequately m 
to this data coil 



to thfe student and employer follow-up 
/meed Wt this 'strategy will not pro- 
|id or reliable data. 
|al concern is the massive potential 
|f the system. The cost' of VEDS imple- 
estimated in some quarters to exceed 
dollars annually. This figure may not 
|asure thi^cal institutional commitment 
Iction task . .• . 

,nH^Q o"i-hopeltha;t {the House Subcommittee on Elementary 
and Secondary Eduction] will address these inadequacies i^^ " 
the design of thelcurrent VEDS. Any such system should in ' 
fact provide valif and usef^il data and should, in addition 
ni? i"s^it"tions and states ^obiigated to 

collection. The^ necessary modificatioii^ should not 

"•"""cragJreTaX'was SrieoSJ.^J -\^£^°"'^1 a^.re.ate.data. 
.1* \c , , i requi.i'ed. the expense and diffi- 

, culty of developing .standardized natit>nal data elements would 

' tLt vTrS"'"^- -r'r '"'^^ modification it d6es no? appeal 
. . that VEDS can se|ve the purposes for which it was established 
and will continu| tq impose severe burdens upori the educa- ' 
tional institutions, (pp. -540^541) ' ^ 

'In addition to toucLng on th? current status" of evaluation Id data 

gathering on vocational Lucation, the 1980 Hearing, also touched on the quest 

of what criteria might |e most appropriate in evaluating the effectiveness 

of vocational education programs. In his testimony, for example. Taylor . 
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mentioned several different measures of effect iveness-^-earnings, employ- 

ment, job satisfaction^ and employer satisfaction, Taylor was asked by 

*Carl Perkins, Chairman of the House. Committee on Education andXabor,. ^ 

which of, these measures he feels is most ^important in judging the success 

of vocat-ion^l programs. - Taylor responded as follows: 

It is extremely difficult to assign arhigher weight >, . 

to any of the criteria fot, measuring the effectiveness 
. of vocational e'ducation. Eamingsv employment , job 
satisfaction and employers satisfaction. 'In an absolute 
sense, however, employment is probably the most funda- 
' mental measure, followed by earnings. ""It also follows 
that individuals who have completed vocational education 
need to b*;satisfied with the job and employers need to 
^- * be satisfied with the performance of vocational graduates^ * ^ 
I beHeve that the four measures tend to be ^cluster, which 
^ arq extremely- difficult to separate and which in fact do 

interrelate and support each pth^f. (p. 331) * . • ^ 

4 

Similarly, Dr. Henry David, Director of the Congressionally mandated Voca- 

.tipnal Education Study, was asked by Congressman Arleji Erdatd'', "How do you 

measure the success 'of vocational education^" He responded as follows: 

I may. sound as if I am hedging, Mr. Erdahl, but if 

you will give me the criteria. Twill tell you how we. 

measure success. - , 

I would measure the success of the program— which is 
not a single program, as you know, it is a multiplicity 
of programs--by the stated objectives of the successive 
, ^ pieces of legislation which how represent Federal policy^ 
^ ^ These build upon, as you are fully aware, the legis- 

lation of 1963, and even a superficial look at that 
structure of legislation would indicate that it ha^ a 
variety of^ goals. j ^ ' . 

If you look at the particul^ goals, you would have an 
*opport unity- to ask questions about whether they were 
realized or not/ , 

When^it comes to the emphasis in the legislation upon 
preparation for entry into job^, you have one set of ^ > 
/measures that can be applied. If you raise a question- 
about .the distribution of Federal funds to produce 
• * 'leverage on new fronts in the vocational education enter- 
. ^rise so that the whol^ enterprise might be redirected^ 
you have another set- Qf measufes. (p. 537) . 
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' Similarly,' George Cop a and Gary Forsberg of the Minnesota Research 
and Development Center for Vocational Education offered testimony that "All 
of the purposes of secondary vocational education must be considered when 
evaluating the effects of vocational education at this level; a process must 
be developed to specifically focus on the purpose of occupational explora- 
tion" (p, 401). 

In sum, the testimony offered in the Congres/ional Hearings in September 

f * 
1980 on current issues in vocational education se^m& overall' to suggest three 

general conclusions. First, while the VEDS system -seems^nerally to be 
considered an improvement over past similar evaluation efforts, it also was 
judged to be imposing a considerable data-gathering and financial burden 
on participating institutions (particularly postsecondary ones) and to be 
leading to the gathering of follow-up data^pn student employment success 
which were in the, eyes of some of questionable' Validity ^nd utility* Second, 
vocational education was generally considered to have a multiplicity of pur- 
poses including occupational exploration and preparation for advanced occu- 
pational preparation, as well as direct eifiployment in vocations. Third, and 
in light of such a multiplicity of coals, there appears to have been consider- 
able reluctance a^nong witnesses to give clear priority to any one or' two 
evaluation criteria ,for the purpose of evaluating the success of vocational 
- education programs, 

Abt Study- of State and Loca l Compliance and "Evaluation Practices (BeUke et al.. 

158^]— /■ \ : ^ 

The Decembet' 1980 study b>j^Abt Associated' provides a third souce of 
information on the current status of evaluatiop 'practice with respect to 
vocational education. This investigation was based on "extensive onVsite 
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interviews in IS spate departments of vocational education-, 16 ^secondary 
education institutions, 12 postsecondary institution^ and 10 CETA prime 
sponsor organizations'." These interviews v^ere "supplemented. by an analysis 
of extensive documents iir each of the 15 states visited and a more limited ^ 
set ot documents /from 41 additional states and territories" (p^ 2). 



The Abt.inq, 
in light of the 



iry looked dnto evaluatidn and^data reporting requirements^ 
evaluation requirements of the 1976 Education Amendments, 



and in this regard concluded as follows: 

The most consistently implemented and according to thp 
state agency staff the most important evaluation com"*- 
ponentlin the 15 sample states is the*program review 
process, . . . The major difficulty which states are 
encountering in meeting the requirements pf evaluating 
planniing and operational process (sic) ds finding the 
resources needed to support the program review system^ 
(p. 96) 

• ••j p • f ^ 

Thb next most consistently implemented 'evaluation 
categ^ory is student employment success, particular student 
follqw-up-'rthe ca-tegory which was given greatest emphasis 
by Cbngress. . • The problems which arise in meeting 
federal requirements for student follow-up data appjpar. to 
stemjfrom procedural difficulties of gathering accurate 
reliable data Which are consistent across districts. The 
difficulty in gathering high quality data in turn results 
from poor response rates; problems in estSabli^hing standard 
definitiono of the data^elemehts together with the lack of 
conviction that standardized followrup data are necessary; 
and from the lack of federal fundiiig to ease the burden 
which the requirements impose on state and local resources. 

Formal employer-^ollow-up^ata are collected much less 
consistently than student follow-up data i . . There is 
less agreement among vocational educators that employer 
follow-up, as opposed to student follow-up, is an important • 
ana necessary indication of vocational program effective^ 
ness or that it can realistically be collected, (pp. 96-97) 
* [None of the states in our sample requires^ statewide 

reporting of student achievement data in all occupational 
arieas, though a small number of •states do' have occupational 
proficiency standards in a limited number of occupations. 
(J- 97) . , ' , . 
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Less attention has been giveiy to evaluating the results 
I of additional services to special needs populations. All 

sample states Which have a formal program review process 
j include some itefifs on special* nfeeds populations, though 

the extensiveneys of the'review varies.- In all cases, the 

review focuses on access* to rajcher than results of voca* 

•tionai programs i (p. 98) 

The Abt analysts also inquired iAto the implementation status of VEDS 

4 ' i * / " ' 

" and ''reaclTed the following conclusions:,- 

One of the major problems iiy i-mplementing the reporting 
system has been the difficulty particularly at the post- 
secondary level, in arrivinjg at common operational defini- 
-tions pf data elements . 

^ A related problem is tlie accuracy of ^data entered into 
the system; people are counted in the wrong categories be*- 
>cause of unclear de^initidns and local staff are forced to 
estimate figures under certain detailed breakdowns because 
local .recordkeeping systeps do not provide actual data for 
those categories. 

Another major problem/with VEDS has been the substantial 
burden it has imposed •on/ the state and local agencies 
resppnsible for collecti/ng the required data. Meeting VEDS - 
requirements requires an investment of substantial resources. 
Tbe 1976 Amendmerfts.^Jautnorized funds to assist states in 
5 implementing VEDS., but /the money was never appropriated and 
states have been forcea to absorb the full costs of the 
system, (p. lOl) 

Wentling's Survey of State^Evaluation Practices in Vocational Education, June 
^ A fourth and inost *fecent source of .information on the current status of 

vocational education programs is Wentling^s (1981) survey. This survey, con- 
ducted in spring 1981, employed a survey questionnaire mailed to directors 
of vocational , education in all fifty^states and territories. Fifty of 57 
questionnaires were returned (88% response rite). Thfe four-page question- 
naires sought answers to eight questions ^pertaining, to vocational education 
evaluation, using checklist, rating and open-ended items, Wentling^s data 
showed that state directors of vocational education rated the following items 
as ories they would rely on as indicators of program quality (shown in paren-', 
thesees are the number pf state directors rating each item as an indipator) 



1981 



Employer feedback (47) ^ . " . | 

• Placement level (44) 
Job satisfaction Df grads (37) L 
Quality curriculum materials (37) 

Instructor performance* (36) • ^ . ^ ! ^ 

Student test performance (31) i 

Sentiments. expressed regarditig evaluation cmteria appeared to be' 

- ^ ' I 

somewhat different in response to another question Went ling asked, namely: 

If federal requirements for evaluation were | 
eliminated, what would you choose to evaluate? | 

- \ 

Responses to the items listed were: . 1 . 

. • • Placement (n=43) ' ' 

Planniaig and operational processes (38) 
Services to special populations (36) 
Employer reaction (35) 

Student performance (34) i 
I IVhy services to* special populations showed up relatively strongly as 

a preferred evaluation criterion in response to this question, but "not with 

I I 
^ I 

regard to the previously cited question regarding indicators of progrjam 
quality is unclear.* % 

Despite these apparent preferences for evaluation criteria, it |ppears 

that actual reliance on them as evaluation criteria was^not>et a faif of 

* ^ ' * I . 

iife in many states in the spring of 19^1. For in another question ftfentling 

asked respondents to indicate which eyaluatibn activities were fully! imple-. 

mented in respondents' states. Responses were a$ follows: 

'o Placement level of grads (35) 

o Employex' assessment -of grads (27) 

o Assessment of pla:nning and operational: procedures (21) 



* One hypothesis to explain this anomaly is that apparently service i 
to specia^l pop'ulations was not ^included as a* response option t.o the 
indicators question. * , ^ ♦ 



o Assessment of special needs services (18) 

0 Assessment of student performance (14) 

Went ling *s dat«a also provide a clear indication of why preferenSes 

for evaluation were not fully implemented. ?or in l*esponse to the question, 

"What factors ^tampered the development of evaluation activities in your 

state?," the following items were each indicated by 10 or more respondents: 

J Inaplequate resources (n=26) > 

Lack of .federal technical assistance (20) " . » 

Lack of evaluation, expertise (15) ' 

Lack of guides and books (13)' ' ^ * . 

Inadequate federal guidelines (12) 
Negative attitudes toward evaluation by LEAs (10) 

Moreover, when asked in ah open-ended question what suggestions ^they might 

* - » 

have for writing* new- legislation, the two most frequently offered suggestions 

re the following: 

V Provide more latitude "for states to develop their own 

evaluation around *their needs ^ (n=9) / • • ' 

y Provide funds for evaluation (n=7) ^ 

Also, an apparent reflection oi the strength of dissatii>faction in 
at least some quarters with regard to current evaluation requirements, 
one respondent sugg^ested an action which would certainly be an unusual 
provision fa include in new "legislation, namely "Fire 
developers of guidelines." ^ ^ ^ 

> In conclusion, Wentling offered five specific recommendations for the 
development of federal policy regarding vocational education evaluation:" 

1.^' Continue to emphasize the improvement function of - 
evaluation and further develbp ways of facilitating 
the use of results fot improvement. 



^ 2. Analyze extant research findings and conduct new -i ^ 
research on thg validity of, process measures^ for / 
predicting' products or outcomes. of vocational edu- 
cation. These findings should provide the base pr 

• determining evaluative criteria, 

3. .The' requirements for evaluation procedures shoulld 

* remain somewhat consistent to minimize state and 
local burden and to, provide f(^f the enhancement /of 
'Current ly^used procedures. ^ ' 

* *4. Any new approaches should be tried out prior td the 

• V. . preparation and issuance of rules and regulations, 

5, The use of ^valuation results should receive^ added 

* emphasis in terms of new procedures, staff d^v^lop- 
"ment , and 'technical V^istance* ^(p. 17) 

• \ * 

And^n a more general auggestj-onAwentling advocated that ["it is important 
to strive" £or a certain amount of ^ability of policy" wip respect to 
evaluation of vocational ^plication Cfe- 1'^) • 



\ These few paragraphs have briefly summari^zed recentj.evidence. on.the 
current 'status of evaluation of vocational education. |he obvious next 
question is what should be done in- the future with respect to f edeis^l 



» ♦ 

"■requirements for evaluating vocational education? Howe 
ing this question in the final section of this paper, 1 



rer, before address- 
5t me make a brief 



detour to review other aspects of federal policy an^x e3<perience in evalu 



ating education programs. 

IV. How Are Other Federal Education Programs Evaluatec ? ^ 



\ 



The reason for addressing this quest ix)n is simpl-e 
years, -a tremendous amount of effort has been invested 
evaluations of education programs. While federal reqni 



Over the last fifteen 



in federally sponsored 
rements for evaluating 



other programs are not directly/ relevant to the evaluation of vocational _ 
education, a consideration of other programs and their evaluation require- 
ments, in light' of how they both resemble and differ from vocational edu-_ , 

•• ■ •' ! . 

-cation, will suggest some unique insights on voca'tional Education evaluation 

policy. ~ ■ ' ' 

There are, of course, many different federal education programs whose 
evaluation requirements might usefully be reviewed. Here, howe,ver, I only 
have time to consider two, najnely. Title T of the Elementary and Secondary . 
Education Act of -1965 as amended, and the Education for Handicapped ■ 
Children program (PL 94-142).. There are several reasons for choosing'these 
programs as comparisons, but one obvious and direct reason is that these 
programs represen^ the major federal efforts to serve disadvantaged 
children and" handicapped children, two of the special needs groups whom 
federal- legislation explicitly mandates federally assisted vocational edu- 
cation programs to serve. Another very practical reason for considering 
•these two programs is that they ar.e the two federal education programs 
which prb^ably reach directly into most of the LEAs also served by federal 
vocational education assistance, and therefore most local educative institutions 
which must meet federal requirements for evaluating vocational education 
. programs also must meet the evaluation requirements for Title I and PL 94-142. 
• Title l\. ■ ' , 

. * ' ' v 

Title i constitutes the single largest federkl source of revenue m 
■ suppprt.'of public elementary* and-,secondary^schools\ It provides money 'to- 
countiel and school districts _to improve the" educational achievement- of" 
educationally deprived children living 4|i ?reas with kigh proportions » of 
low income families.' .AS mentioned already, Title. I .of\ 1965. was the first 



major piece of federal legislation to inJiude a ^jrovision for the regular 
evaluation of funded programs. McLaughlin's' (1975J analysis of the legis. 
lative. history of the evaluation provision in ESEA (k sT-lQ) suggests two 
. con^ting visions behil^d the e'valuation mandate. To some, .the evaluation 
prcasion was intended as a tool ,o provide parents with more information " 
about program efforts funded under this title and" hence to enhance the V 
. accountability of programs to the pk^ents and communities' they Served. To 
others, particularly at the federal level, evaluation "was viewed, as.a manag.- 
men^d. planning device for infusing rationality in subsequent program 
efJbrts. As a result there has been' a continuing tension in Title T 
evaluation policy, similar to the tension identified by Hendricksort (1981) \ 
with respect to vocational education policy. Should Title I evaluation 
serve mainly the management and accountability needs of states, the federal 
, administrative branch and the U.S. . Congress, or should it be aimed more to 
serve the information needs of local offi,cials. teachers and parents? 

IVhile the evaluation ^r^vision contained in PL 89.'10 was a compromise, 
according to McLaughlin, intended to address both visions of the main ends 
that Title I evaluation ;was seen as serving, by 1967 the'federa^ interest in 
evaluation as a mana^emeilt and planning tool clearly came to predominate 
in the educational am^dments of^that year (PL 90-247, section 404). With 
the 1967 amendments the Secreta^ of HEW was required to submit each year 
an evaluation report to all relevant Congressional committees, regarding 
the effectiveness of programs funded under Title I ESEA. The report was to 
include a detailed review and ^valuation of programs for their entire past ' 
life, based to the maximum extent pricl^able on Objective measurements. 
The clear intent was to use this xn^matiSn in directing future programs 
decisions at the federal level. 



"s, ■ - . . < ' ' \ ^ 
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The federal need for evaljuaiioi) information was. further emphasized 

in the Educational Amendment of .1974 (PL 93-380} Section 151, which 

authorized the commissioner to provide for independent impact evaluation; 

, , . to develop standards and evaluation models to be used rby LEAs; -to^^t 

objective criteria, * an^'^to develop. a system that would produce data^ that- 

are comparable statewide and nationwide, to provide technical assistance to 

\^ LEAs in implementing the system; and to make au annual report to Congress on 

^the evaluation results. While Section 151 (a) 'of the 1974 amendments 

suggested the inclusiion of parents* a'nd project participants^ opinions in 

J any LEA evaluation, none of the rules and regulations that f^i^owed inade any 

further mention of procedures for implementing this provision. 

The Educational Araendmms.of 1978 (Section 124) saw a re- . 

emergence of attention to the local perspective: , ^. 

(g) Evaluations .--A Ibcal educational agency may receive 
' funds under this title only if — 
(IJ effective pro<?fedures are adopted for evaluating 
at, least every three years the effectiveness of the programs 
assisted under this title in meet-ing the special educational 
nee^s of educationally deprived children; 

, (2) such evaluations w ill i hclude, during' each three- 
year period/^ the collection and analysis "of data relating to 
the degree to which prop*ams assisted under this >title have 
achieved their goals , and will also include objective measure** 
ments of educational achievement in basic skills over at least 
^ a twelve-month period in order to determine whether regular 
school. year programs have sustained effects over the sunnner; 
^and , 1^ . . ^ 

• f3) the "evaluation WdT address the purposes of the program 
and tlmt the results of the .evaluations will be utilised in ■ 
planning for and improyilig projects and activities carried 
out under this title in subsequent years (emphasis added)* 



:ERic; 



^ By 1980, the mandate for the Commissioner of Education to develot) 
standards and evalyation models for" Title I evaluation had evolved into 
a complex* system* under which each LEA receiving Titl'C I funds is required 
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to evaluate Title I programs at lea^t once every three years, usin/reliable 
^ and valid instruments, procedures that minimize\rror. and yields a valid V 
assessment of achievement gains . Toward the latter end:the USOE sponsored 
tHe development of three alternafivV evaluation models ; .J- \ ' 
; (the norm-referenced comparison poup, anU regression group" . 

models) for estimating the achievement gains of children" served in Title i 
programs in the areas, .of reading, math and language.' Each model is supposed 
^ to provide an esUmat^e of the effect of children's receipt of Title I ' 
- services ip comparis^o what participating chi^dre^ migh't have achieved 
in the absence of Title I services.* According to most recent evidence " ^ 
' (Anderson et al. , 1^)78) .ost LEAs are using the norm-referenced model 'in' 
order .to meet federal Title I evaluation requirements . Achievement gai.s • - 
estimates derived from LEA's applications of the three models (or at Icasf 
in principle, any alternative model for estimating gains tlat is approved • " 
-by the Conunissioner) are to be. reported to SEAs using a c6m:.on metric, the " 
normal curve>quivalent or NCE. Each SEA can thus aggregatS results of-/ • • 
^ local Title I evaluations to report to federal officials who ift turn can' ° 
produce national level aggregations of the .estimated. impact of Title I 
nationwide.^ B*carl^se of the complexity of implementing' the Title I evalu^"^" 
^ation requirements and mpdels, the Education Departm^^s instituted ten ' 
regional Technical Assistance Centers to help states and^d ties! implement 
Title^I evaluations. ' ^ 



It should be noted, however, that seve^Val evaluation experts and some 
empirical research has questioned the extent to which aSeJeLnt a|?Js " 
estimates derived from these models ar« comparable-, - * - 
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Education for All Handicapped Children • j 

lo Noveinhe;r 1975 > the U^S, Congress passed the Education for All 
. Handicapped Children Act (PL 94^1421 mandating that by September, 1978, 
all school^aged handicapped children in the United States be assured "a 
free appropriate public education/* Federal evaluation requirements im- 
plemented in .connection with this apt are markedly different than those' 

^ prertaining t<> either Title I or vocational education, not only becaus0 

/ ' 
. tbe Handicapped Act is of more recen^ vintage than federal initiatives 

/' ' 
witb respect to vocational education and education for disadvantaged child- 

' i'' » 

rent, but ^Iso because it embodies a quite 4ifferent philosophy and approach 

/" 

to educational programming than either Title I or the various pieces of 
federal Vjicational education legislation. 

[• - ■ • ■ ' • 

The /I275 Education for All Handicapped Children Act specified a r^nge 
of activities that school^must conduct with regard to handicapped children, 
activities that are not viewed as a unitary program, but instead are seen 

/ ■ ' ■ ' ■ 

as procedural insurance that handicapped children will benefit from the •* 

• /' ' 

rights/ they have been guaranteed under the. Apt and individually prescribed- 

educal^ion. 1 

[The Act] requires, that specialists be calleti upon to evaluate 
the children's special needs and "determine the mosTt appropriate ^ 
educational envirorunent for these children; that- an individualized 
education program be developed fro each child identified needing 
special education or related .serviC*^; that the schools notify par* 
ents of findings concerning th^^ftgfilAldren and include parents in 
the process of making decisions regarding how and in what circuits tam:es 
their children will be .educate^; and that an opportunity for a hear/ng 
be provided to a parent who is dissatisfied with the schools* de- 
cision,' Further, the* Act a^ks Ithat, to the extent that.it is in 
the child^s be^t i'ntere$t^ jfcachl handicapped child be educated with 
nonhandicapped children t (Bureau of Education for the Handicapped, 
1979, p, 71 ' ^ * • '-i^ 
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According to the first annual re'port of the USOE's Bul'eau of Edu- 
cation for the Handicapped on implementation of the 1975 Act, federal 
contributions (under PL- 94- 1^42 Part B) amount nationwide to about 9% of 
total state and federal funding of education of handicapped* children 

(Bureau of Education for the Handicapped, 1979*,. p. 214). Thus the federal 

♦ 

share of funding for the handicapped appears, to be roughly equal to. the fed- 
eral share of funding for vocational education — according to the most 
recent data frbm NCES, federal expenditures provided abolit,8.5 percent of 
total vocational education expenditures in 1979 (U.Sy* Congress, 1980, p. 89). 

So what then are the federaj requirements for the evaluation of 
federally- aided education programs for handicapped children? To put the i 
matter baldly, how do the federaF evaluation strings attached to funds for 
education of the hancficapped compare with the evaluation strings attached to 
federal funds for vocational education? The Boruch and Cordray study de- 
scribes federal evaluation r.equirements for LEA's under PL-94-14'2 as follows 

At the local level, the term evaluation refers primarily to diag- 
.nostic assessment of children. The regulations require that pre- 
' placement evaluation be conducted using multiple, appropriate 
assessment modes. If the child i^^-£Qund to have a handicapping 
\ condition, an Individualized Educational Plan (lEP) is .devised. 
The content of the Indvidual Education Plan i^- required by the 
regulations to include: (1) an assessment of present- levels of 
educational performance; (2) a statement of annual goals and short * 
term instructional objectives; .^(3) a statement of specific special 
education and related se^^dces and an assessment of the extent to 
which the child is able to^articipate .in regular education programs; 
'(4) projected dates for inititiation and termination of services; 
(5) appropriate 'objective criteria, evaluation procedures and a 
schedule for r^evaluation. (Boruch and Cordray, 1980, p. 3-10). 

Obviously- these evaluation requiremeifts are quite different from 

*In citing these dat| regarding^fede-ral contj^ibutions for the handicapped 
and vocational education, it should be nptedthat from the sources 
' cited it is not altogether clear whether federal contributions as a^ 

share- of total contributions have been calculated on exactly com- 
" ^arable bases. Thus the only concjji^ion that seems warranted is 
that federal contributions to education for the handicapped and to 
vocational education seep roughly comparable -.on the order of 10% 
of total expenditures. 

. ... - -as 
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those mandated under eitl^er Title I or federal vocational education ^legis- 
lation. The/ pertoin exclusively to, procedural requirements and do not 
encompass any sp^R.fic outcomes such as -student achievement gains or ^ 
employment success of handicapped children after leaving school. There 
appear to be two broad reasons for the difference: the first having to. 
do with, the assuUmptions about the relationships between federaf, state, and 
local levels of educational authority, and the second relating to assump- ■ 
tions about educational programming and children's individual educational 
' needs ► ^ . 

In her account of the process of federal efforts tl) develop an eval^ 
♦ • ** 

* *• 

uat ion! plan, for PL-94-142,' Kennedy mentions this Act ^as unique because it 

"more clearly delineate|iyie relationship between federal , state, and local 
agencies'* than prev*bus federal education legislation (Kennedy^ 1978,. p. X9), 
The essential idea was that states would have primary responsibility for 
implementing programs to insure children's rights under the law, and that 
the federal role was one of oversight responsibilities with respect to state 
activities, Iif the process of developing a plan for. evaluating PL-94-142, a 
decision was reached that the primary audience for federal evaluation activ- 
ities should be the "federal agency and the Congress" (Kennedy, 1978, p, 37)T 

^Evaluation requirements in connection with PL-94-142. also were in- 
fluenced by assumptions about the nature and appropriate level of decision- 
making regarding educational programming • One of the key ideas in PL-94-142 
was the individualized education plan or lEP* Again Kennedy's aceount provides 

• insight into some of the thinking behind this provision of PL-94S142: 
/ * . 

THe Act implies a faith that those closest to children -- their 
teachers and parents Will make the. best decisions about children. 
The purpose of jthe Act -•is not to increase the academic or social'^ 
growth of handicapped children but rather "to provide a free, 
ppropriate public education" to all handicapped children. 



Mdst education programs have several goals, many of which 
may not even be related to the children — for example increased, - 
efficienty in service delivery, fairness in assessment, equitable 
opportunities, improved parent satisfaction, or simply improved 
accountability* This ^ is not; to say \hat academic performance 
should be ignored but that evaluations, of these aspects of educa- 
tional programs may be more useful locally than nation^-Hy, The 
local area has more capability to modify and improve these aspects* 
of programs which affect children's performance- in educational. 




a result of such thinking, fjederaP evaluation retjuire- 
ments developed for PL-94-142 specify no particular outcome measures for 
ti>e^ purpose of evaluating education programs for the handicapped. Instead 
federal requirements focus exclusively on procedural requirements; concerning 
diagnostic assessment, development of lEP's and guidelines on who should be 
involved in the development of lEP's-. ^ , ^ 



it it * * it it " ^ 
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This brief review of federal evaluation reqi|^rements'' developed in 
connection with' Title I and PL-94-142, anS how they compare with requirements 
for the evaluation of vocational education, raises interesting questions* 
What is the appropriate balance between procedural requirements for the pro^i^^ 
vision of educational oppprtuniti^e^ , the attainment of certain outcome^ in 
Educational programs, and the consequences for individuals\^in^^ after 
schooling? What is the appropriate balance of activity and responsibility 
between the federal, state, and local levels of educational governance? 
There are I think no definitive answers to these questions, but this brief 

• review of vocational education," education for disadvantaged children, and 

. ^ ^ 

education for handicapped .jritildren should make it clear that the evaluation 
requirements connected with federal initiativtes in these three areas avre 
premised on some rather different answers to such questions* Why this is so 



is a question worth scxnitiriy, ;I think, but for the present let me reiterate 
• ^ I I ^ 

only that to the extent that local educatioftal agencies meet the mandate 
for vocational 'education^ to' s^erve disadvantaged and handicapped children 

. they appear to be required to meet some quite different maptlates for evalu- 

^ ating the services provided. - 
V. Federal. Legislation; What Criteria^ Should Be Set Out^ for the Evaluation 
. of Vocational Educaition? 

As the discussion in,^the last section makes clear, answers to this 
question depend upon somp rather fundamental assumptions about the appropriate 
balance of authority for" vocational education among local, state, 'and federal 
levels of jgovemance. While. I have no answers beyond personal predilection 
a ' , regarding what such assumptions should be, I think it clear that when it comes 
to establishing evaluation criteria the fcdcrAl role, probably ought to be, ' 
given current circumstances, relatively minor*, Afte'i' ajl, the federal con- 
tribution to the funding of vocational ediiqation ijationwide appears to be no 
more'^th^di -10% or So of the tot^l effort- In tHis light, and because of the 
concerns of '3tate vocational offdcialfe, n^Vfewed^^in section III, ii seems 
reasonable. that' either the federal mandate foj .evaluation of vocational education 
, should be lessened or federal funding should be provided to cover a sub- 
stantiai portion of the costs *o£ meeting the federal evaluation mandate* At 
the same time^ how^-vier, ;1*ent,ling*s advice, ^namely that whatever happens the 

federal govemftent should -strive for % certain d^r6^ of stability in terms ^ 

' - . ^ - *) . • . 

• of vocational education evaluation policy^ After dXi the evaluation reqiuire- 

'^ents set-out in the Education Amendments of 1976 and elaborated in regula- 

- - . ^ • i • • - , , • 

■ • . * tions^ in* 1977- and in the 1979 polic;^ memorandum' are really only beginning to 

be implemented* ' * ' \ . 



Purjoses of Evaluation^ One other general consideration concerning 



the puTp3ses of evaluation is also worth getting out before discussing various 
types of evaluation criteria, As the review of the development of current 
evaluation requirements for vocational education (in section III), the brief 
discussion of evaluation requirements for Title I and PL-94-142 (in section 

I\Q clearly suggest, one prime issue in the establishment and evolution of' 
federal eivaluation requirements has been the question of purpose: what ends 
and whose interests ought evali^t^oif^to^ aimed at serving? ^ In terms of 
federal mandates for evaluation there has Been a continuing tensi'on between 
what might be^called national interests and, local interests. On one hand 
^evkluation has been intended to^produce valid, accurate and , comparable data, 
so that evaluation results can be aggregated from local to state levels, and 
from state to the national level to produce an overall view of particular.^ 



programs. Of the thre^ programs whose evaluatjLon requirements have been re- 



viewed Title I seems to have adhered mo,st to this view of the purpose of" 

' ■ • ' €^ 

evaluation. . ^ ' ^ - l ^ 

- Yet on ^he other hand evaluation requirements for all three programs 
(including as noted in section IV, increasingly in Title I) there has been 
the view that ^evaluation ought to- provide locally relevant information which 
will be of assistance in improving prbgrams at the local level regardless of 
whether or not ,the evaluation data are ?ufficiently standardized to allow 
^gregation beyond *the local levels Amohg the three programs reviewed, this 
tendency seems, to have been most preeminent with respect to PL-94V142, ^ 
What then should he the balance between these competing interests with 



respect to vocational educatioa? Clearly it seems that the local program 



improvement goal should have 'highest priority with respect to vocational* 

education evaluation. First, as Hendrickson (1981) points out this seems 

to have been the predominant .goal that Congress had in mind in writing 

evaluation requirements into the Educational Amendments of 1976 (even though, ^ 

•as Hendrickson also points out, this goal seems to naj ^e been somewha t^i^s^ghted 

in the process of developing regulations for vocational education evaluation) • 

Second^ this view seems clearly to predominate -ampng vocat;ional education of- 

ficials, as noted in the review of evidence on the current, status of vocational 

education in section III. This suggests at a minimum, whatever criteria 

— - I » 

are established for the evaluation of vocational education in federal leg^s- 

latibn, if their main purpose is to serve local p^ogr^ improvement efforts 
that they need not be implemented in a standard way so as to provide nationwide 
.comparability and, to allow aggregation across^ the diverse types of institu- 
tions which provide vocational education programs at structurally different' 
levels of the nation's educat^iona^ system. In this regard, the Title I 

evaluation and reporting systems (TIERS) provides a sobej-ing contrast with the 
Vocational Education Data System (VEDS) • TIERS focuses essentially on three 
types of outcomes (namely reading, language, and math> achievement) served by 
Title^I programs in institutions which are relatively homogeneous (namely 
local eclucation agencies); Yet even so, the effort Jo produce a national pic- 
ture of Title I through TIERS has. received some hard methodological 'criticism 
(see, for exaii5)le, Jaeger, 1979;'Linn, 1979; and Wiley, 1979). In contrast, 
consider how much r.ore ambitious VEDS is than TIERS. Vocational education 
serves students in institutions of , post-secondary education .as w6ll as ones 
governed by local education agencies. VEDS aims at gathering data. not just on 
student achifevemeilt in basic academic skills^, but als-i) in occupational com- 



petencies, and followr^up data on eiaplpyrient succe§$ defined in Pterins of 
placement and employer satisfaction, Moreover, despite the ^relative simpli- 
city of T-IERS in .contrasi;t to VEDS, tfie^Department of Education has ^established 
a system^f regional technical' assistance centers (TAGS] to help state and 
local educatiort^ag^cies implement .tTie Title I 'evaluation and reporting 
system* No'similar\ystem hajs been established to provide ^technical ^ 
assistance with .respect U^vocational edjucation evaluation and repairing** 
In sum, then, it seems tO\me that- there are several dif f^xpnt grounds 
for concluding -that ,the cuinrent fedeAl mandate for evalj^^ting vocational 
education is overly ambitious: ' ' 

-In' the testimony of state vocational education officials, 
• the current requirements impos'e bpth reporting and financial 
burdens on agencies implementing vocational education pfbgrams 
with federal support* • ' ' . , t 

-In comparison to federal evaluation and. reporting requirements 
for Title I and PLT^94'^142, the reqi)i^rements- for vocational -edu- 
cation are far more extensive, yet at the same time far less 
fully supported^either in. terms of funding or technical * assistance • 

*In the apparent ijitent\ of the M.S. Congress in writing evaluation 
requirements into the Education Amendments of- 1976, and in the 
clear opinion^of state vocational education officials, the major 
purpose of vocational education evaluation is program improvfement 

'at state, local; and institutional levels* JJonetheless, several 
pf the evaluation criteria currently maijdated for vocational 
education ab:e of questionable ut;Llity in serving that end* 



* It should be noted that' the 'NCES does provide. some limited support to 
state officials regariding the implementation of VEDS and vocational 
education evaluation* For sample, NCES does provide financial support 
to cover travel costs for state official^, to gravel to other states to 
inspect their vocational education evaluation procedures. 



Given^is overall conclusion, what more can be 'said, about specific 
types of eyalyation requirement's? Answers to this question are organized 
around Ithe four fypes of evaluation criteria mentioned in the 1977 regula- 
ticjns knd described in Table 1, namely, planning ^d operational processes; 
studerit achievement; employment success; and services to special populations • 
Sincelthe employment success , criteria were ones given special emphasis iti 
% the |976tAmendments they are discussed first, * ^ t 
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Eittpioyment Success 

As mentioned in the introduction, the 1976 EducatioA Amendments ex- 
plicitly called for the evaluation of occupational vocational education in 
terms of the extent to which progr.am 'completers' afid leavers! find employment 
'in occupations related to their, training .and are considered Iby their employers 
well-trained and prepared for employment. In tffe 1977, regullp-tions^ the .follow- 
• ing measures were itsted as appropriate indices of employment success: 

(1) Rates of employment and unemployjnent; 

(2) Wages rates; ^ * , . 

(3) Duration pf employment; and ^ : ^ ' i 

(4) ' Employer satisfaction with perf6rmance of vocational education^ 

students as conipared with performance of persons whp have not 
had v9ca,tional education/ . ' . ' 

First note the differences between the legislative and :i!egulatory 

language^. Though the 1976 legislation mentions "employment related to • • < 

training/' the regulations speak more generally about rates of eraplo/ment/ . I 



am not sure whether the shift in emphasis was intended or not. The literature 
on vocational education since 1976 still contains fairly frequent references 
to emplojnnent related to training. Nevertheless, de-emphasizing the "related 
to training" issue as part of a standard evaluation criterion seenls eminently 
sound to me. Why? Because there are numerous different way§ o£ d^ermining 
job - training relatedness and different procedures can yi^ld quit\e different re- 
sults. While many follow-up' studies on vocational education particljlpants ^ 
appear to rely on subjective judgements of job relatedness (usually ^as judged 
by teachers or former students) at lea$t some, studies rejy on more systematic 

'probedures for determining job [relatedness to training. One study evaluated 
, * ' * ' \ 

relatedness by^ systematically analyzing job titles in comparison to vocational 

- ' ^'^ . ^ ^ 
education specialty area and found that there v/as only arouiid 55% agreement 

. 4 * 

I - ' " 

between results of the systematic analysis and. teachers, judgements of job 
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relatedness.* This suggests clearly that basing eval- 

uations of vocational ejiucation on subjective judgements of job relatedhess 
places the endeavor on a very^ shaky foundation. While problems of sub- 
jectivity seemingly might be overcome by development and dissemination of 
standardized procedures for determining job-relatedness and for dealing with 
other practical probleips as well, there are more fundamental problems, \ 
discussed below which suggest that this may not' be a reasonable strategy, 

^ A second type of indicator of employment success mentioned in both 
the 1976 legislation and in^the 1977 regulations is employer satisfaption. 
Note, however, that there is a significant difference between the language 
of the two,. The legislation referred to the- extent to whith former students 
are considered by their employers "to be well-trained and prepared/'* while 
the regulation:; mention "employer satisfaction with performance of vocational 
education students," I have no special insight as to'how or why the legis- ^ 
lative language was transformed into the regulatory language, though I can • 
imaging several plausible reasons for the change for example, it might 
have been due to the supposition that while employers might have a hard, 
tiwe evaluating the previous training of their employees, they would be in a 
much better position' to evaluate their employees* current job performance, 
performance which presumably reflects previous ^training. But/whatever the 
cause for the switch in emphasis, it seems clear that it has caused some 
rea»l problems, App^arently the focus in employer follow-up -surveys on 
individual's work performance (rather than, on their p-reVious training) 'has 
caused concerns about implications for the privacy rights of indiv^i(fuals, ■ 
According to th^e Abt report, some states have judged that privacy rights 



*See Vrpo<3s4incI Haney, 1981, section 4,S for a fuller description of this 
•study. 
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require that the permisision of the former vocational education students 
and/or their parents be obta:ined before conducting follow-up ^surveys with 
the former students' current employers. Thi.s .causes problems of both admin- 
istrative burden and of inference* If some individuals do not give permission 
for their current employer to be surveyed nothing at all can be inferred about 
those individual's vocational educatipn programs. According to the Abt report 

One state in^our sample* feels that the need to obtain permission 
o before contacting employers prohibits the' administration of em- 
ployer follow-up at the state level (Beuke ^ et al, 1980, p. 81). 

Obviously such a conclusion, even in one state, must raise doubts about. the 

appropriateness o^^valuatiqn criteria which cannot be used ^because of* con- 

cems for' the privacy rights of former vocational education students. 

Two other indicators^of employment success were mentioned in the ,1977 

0 ♦ 

regulations, jlamely.wage rates and duration of employment. It is easy to 
imagine why -such criteria may have beefi^ suggested, l^age rates are, for exam- 
ple, one pf the most widely recognized indicators- of economic value.' Indeed,; 
classical" econpmiic theory would have us believe that wage rates in a freely^ ' 
competitive market are the only means available for comparing the value o^ 
different kinds of ]febor. And attention to duracion of employment woiild 
seem to be a natural outgrowth of intention expressed in federal legislation 
as long a^o as 1963 for ^vocational education to help individtials achieve 
employment stability (see p. 7). Yet using wage rates and employment dura- 
lion as evaluation indices obviously raises numerous practical problems. 

■ Nevertheless, even putting aside the. practical problems of relyinjg 
■> . ^ 

*With respect to wage rates, -for example, \he Depafrtment of Labor (1980) 
points out that "Earnings ih an occupation vary by^^eographic 
region. In addition, workers in the $arae occupation may have 
different earnings depending on the industry in which they work'^ 
(p» 6). Such factors' obviously make wage rates a highly imperfect 
' . indicatoir of the quality of occupational preparation* v 



on any of these indicators of employment success as criteria^for evaluating 
the effectiveness of vocational* education*, it seems to me that there are two 
.much larger problems vith the entire logic behind reliance on any one or any 
one set of indicators of employement success. First, it is significant that 
substantial knd apparently increasing proportions of secondaiy Vocational 



education students do not cease their formal education upon completion of . 
their high school careers. As noted already, according to, follovy-up data on 
the high school class 6f 1972, more than 50% of the students self- reported 
in their senior year of higfi school to be enrolled in a vocational program, 
undertake at* least some form of postsecondary education within four years 'of 
high school graduation. Moreovesr, in the latest available report from NCES, 
focusing only on occupational v9cationaL educatian at the secondary level, it 

£^ J ^1- ^ ^ - " 

Wcx^y xutriiu uiidL. ^ 

^ . ' • % ^ - 

1 Of students completing vocational program requirement's, slightly 
more than half are avallat^le for immediate placement in the labor 
force. The proportion ha&dr^pped slightly since fiscal year 1972, 
from 58.9 to 55.2 in. fiscal year 1978 (O.S. Congress, 1980, p.86)^ 

If one is interested in indicators of employment success- it is of 

'Course reasonable to focus .^tt^ntion only upon those who are in th^ l&bor 

force, or in the words of the NCES report, are available for job placement.; 

Indeed, such a delineation ^bf attention with respQCt to measuring employment 

succ e ss is almost r eq uired i -j^ flight o^^. standard Department of Labor prcy:e-- 

dures for Calculating indii^es ^i^K ^s labor force participation and unemploj"- 



* 



ment- rates, Nevertheless, from the perspective of evaluating vocational 
education programs r that^is acti\dties designed to answer questions. about what 

,a program does in the interest of making judgeijients about the program - focusi 

' ^ . .... ^ • . > . 

*rn this regard, it should .be ^noted that calculation of such commdhly known 

economic indicators^as unemployment rates is not as straightforward as might 

be assumed*. In reanalyzing national longitudinal data sets, we have found 

(see' food's and Haney,. '1981 , chapter 4) that^^it is virtually impossibly to 

reproduce DDL procedures pr^ci^sely unless one has access to the same set of 

-questions asked. in DOL purveys,. , - . . 
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heavily or exclusive!^ on the employment success of former vocational 
education students tends *to ignore the 40 to 50% of secondary occupational 
.vocational education program completers who are not known to. be immediately^f 
available f6r job placement. This point is especially crucial in light of ^ 
the Congressional stipulation in the Education Amendments of 1976>4hat ''in . 
no case can pursuit of additional education or training by program com- 
pleters or leavers be considered negatively in these evaluations^i^of programs 
tpurporting to impart entry level- job skills (Sec. 112, b, 1)*. 

This clearly suggests that follow-up studies on former vocational 
education students - including those in occupationally' specific programs - 
• ought to be Construed mojre broadly than *as effort-s to .determine the employ-^ 
ment success of« those in -the labor force. Instead,, in light of *th^ apparent 
tendency for sccondar>' vocational educate. on students increasingly to pursiie 
other activities (including further ediicatiori; -Oc^^jJat^onal ^training, and 
*h'omemaking) , it is clear that to afford a fuller picture, bF^a^# vocational 
education programs do', follow-up studies should encompass T desci;iption ♦of 
what program completeres and leavers are doing b^ they in o\^t of ^the 
labor force. , . ^ 

Despite this recommendation for broadening the scope of fotlow-up 
studies on the status, of vocational education program .coinpl^t^'rs arid leavers, 
a rtote of caution should be interjected here regarding the likely utility of 
any kind o^ follow-up^ studies. As mentioned, the major purpose of vocational 
education evaluation - both in the apparent intent of Congress in writing the 
evaluation provisions of the 1976 Amendments and in the visions of 'vocational 
education officials surveyed in the -studies summabrizeB in section III - is 
iroprovem^t ^of vocational ^ograms at the state, Ideal, and institutional 



problem in this regard is one of timing. Judging from previous 
evaluation reports, it seems not unreasonable to assume that follow-up 

* / ' s 

e 

Studies on the Jost-program status of vocational education program completers 
and leavers take a full two years to qomplete - one year for individuals to 
enter the labor market or into whatever other activity they might undertake, 
and one year for the actilal conduct and reporting of evaluation activities. 
Assuming that program managers do use evaluation findings to* improve voca- 
tional program offerings, it seems reasonable to assume that another two 
years will elapse between evaluation reporting "and the actual implementation 
of improvements based on t ho^||^indings - one year for considera*tiori and 
planning based on the findings and another year for developing and implementing 
improvements in terms of staff training, curriculum development, etc.* 

Moreover,^if the vocational education involved has a two year course 
.of study, another two years will elapse before graduates from the revised 
improved program will graduate and be available for job placement, advanced 
training, or whatever. Thus it seems reasonable to assume that at least 
five years will elapse between gathering follow-up data and completion. of 
the cycle, of program improvement and graduation of students from the new im- • 
proved program. Obviously then, there is a real limitation in this approach 
as a means^ of keeping vocational program offerings current With labor market 

trends, for labor markets can change substantially within a period of five 

, , <. 

years. • , , ' " " ' 

. ' , ^ <ji 

There is. onfe final, and 'to my mind, most severe weakness in the idea 

■ ^ - • • -C . ■ • . 



^This time projection is ^according to some recommendations, unrealistically 
short fdr program development and implementation. Weikart and Bi^net (1976),. 
•for example, argue that it can take as long as ten years ^to develop, test out, 
and .implement new educational programs. 



that follow-up surveys of the employment success of former vocational 
students will help" make voc6lional« program surveys responsive to labor 
market\ demands. Follow-up surveys can help to determine- the ^ployment 
success of completers and leavers^*||^programs that are currently offered, 




but such surveys have little potential for identifying job openings and 
thereby potential jfrogram offerings that are new. In other word's, follow-up 
surveys may provide data on the effectiveness of past program offerings, but 
have little potential for directly illuminating potential fyture offerings. 

In arguing against the liKely utility of follow-up employment success 
surveys as a means for keeping vocational e^diication program offerings current 
with labor-markets, 1 should note that I am ^ot a^rguing Sigainst the general 
utility of follow-up surveys. Indeed, experience in educational research has 
clearly shown that ♦follpw-up surveys as long as 20 years after prograj5r<SDm- 
pletion can be, of considerable value in i"nf luencing general thinKong on 
education and general strategy of program offerings. Rather, I am simply 
noting the likely limitisd utility of follqw-up surveys in meeting the shorter 
terra goal of keeping vocational program offerings curr^t with labor market 
demands. - . . • 

Student Achievement ^ ^ * . * ' * 

As noted in the, reviews of studies of vocational education program 
evaluation in section III, this appears to be the area in which prbvisions 
of the 1^77 regulations are least fully implemented. Recall that in this-- 
area thi^ regulations suggested. reliance on indicators such as: 
-standard occupational, proficiency measures; 
•^criterion referenced tests; and ^ 

r < 

-Other examinations of students* skills, knowledge, attitudes and 
I readiness for entering employment successfully. 
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It is easy to understand-^Why these provisions have not been widely 
inlrpleinented. Standard occupational proficiency ^peasures simply do not exist, 
fir most occupations. Criterion referenced tests are ^^^latively new idea 
In the testing ]^rofessioTi and the technology of criterion referenced testing 
Ls only beginning to be developed.^ Moreover, there is considerable un- 
certainty about what factors are significant determinants (be they skills, 
knowledge, attitudes, connections, or whatever) of individuals* successful 
/ entry*into employment. For example, one recent study of the labor market 
,for male youth with less than a college degree argues that the youth labor 
market has structural features which "ate not widely understood, much less^ 
reflected in OTployment and training policy. # 

Therefore as a gneral matter it would seem^ unwise for federal mandates ' 
to require any specific or standard approaches to measuring student achieve- 
ment in occupational proficiencies. Nevertheless/ there appears to be one area 
of student achievement that may be worthy of more at't^Jl^^on, namely student 
achievement in the basic skills of reading, writing and mathematics. Im- 
parting such skills is not traditionally seen as a priority goal of vocational 
/education, but there are several reasons for concluding that more attention 
\o basic sRills achievement of vocational students is warranted. First, there 
has widespread concern nationally over the last four years or so about 

* S^e BdrJc, 1980', for an up-to-date assessment of the state of the art of . 
crij^ion referenced measurement* 

# 'TKe study entitled Gglting Started^ iis by Paul O^terman (M.I.T. Pfes$, 1980).* 
While Osterman^^s' work cannot be summarized here, let me note simply that 
the author argues, among many other points, that **much of the poljLcy focus 
oh improving the ^school to twork transition* is misguided*^ (p^ 154). He 
Argues this b^^cause of evidence that male youth often .undergo a period bf 
employment instability, a period, of settling dowii, and because "there is 
'./little relationship between tlie first job. after school and later jobs'* (p. 154), 
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the basic skills of all high school students, including college preparatory 
and general as well as vocational program students.' This concern has been 
manifest most clearly in the minimum competency testing movement. * Since 1975 
more than half the ^^^^^ nation have initiated minimum competency 

testing programs, for all public school students.- Under such programs test 
results are use^'to determine award of high school diplomas, to controj grade 
*to grade prpmdtion, and/or t;o identify students to receive remedial instiruc- 
tion. ^st such programs schedule testing in the high school. years, and test 
reading, math, and to only slightly lesser extent writing tGorth, et al., 1980 for 
^iraetails). While many observers have questioned the likely efficacy of.mini- 
. mum comjSetency testing as a strategy for improving learning, it seems undeniable 
that such testing reflects a widespread concern for the basic skills achieve- 

9 

*-ments of high school students. And as a result, all high scliool students in- 
cluding college prep, general, and vocation piogram enrollees (but in somp 
places excluding handicapped students) will have to meet common standards of 
performance on basic skill tests. 

Second, more att-ention to general basic skills seems warranted in . ^' 
light of the tendency doqiamented above for substantial proportions^ of secon- 
dary vocational education program completers and .leavers tc pursae post-- 
secondary education, additional training, -or other activities prior to. entering 

the labor force. In short, for the 50% or so of secondary vocational students 
<^ % ' * * 

who are not imflied^tely avajilable for job placement after leaving fheir voca- 

•tional programs-, basic skills- may be a more general and relevant type of' 

achievement to evaluate than specific occupational competencies. 

Third, even for vocational students who go directly into the labor ^ 

market, basic skills raa^ be more important than ^oc,cupationa*lly specific com- 



petencies. Why? One prominent economist, Eester Thurow, argufes this point 

as follows* First he maintains that ^most skills in tbe U.S. are not acquired 

in formal education* or training; rather they are acquired through on the Job 

training from one worker to anothei'HThurow, 1979, pp, 324-325). He cites 

some "very clear" evidence to support the proposition. In light of this 

proposition, he argues that the prime purpose of vocational education ^ought 

to be creation of background characteristcs which tend to make individuals 

into low-cost trainees . for potential' employers. And from this, he argues 

third that "tJ^e three R's (reading, writing, and arithmetic) are an important 

ingredient in training cost?" (p. 327). He also argues that good work habits ^ 

are a second general type^ of characteristics* which make' individuals attractive 

to employers as pptentially low-cost trainees. Also it should be -noted that 

uiiere are xiupoxLa.XAu quaixo^icauxons to ^nurow S' ^en6 rai tram Ox xeasonxng . 

wlvich for the sake of brevity will not elaborate here. Nevertheless, it 

seems €air to summarize his general point as follows. Since employers tend 

not to sSlj^ct employees on the basis of theii^ occupjitionally specific ^com- 

petencies, but^ instead on the basis. of their trainability, basic skills 

attainment may be a more'',inq)Qrtant goal for vocational education students 

planning to enter directly into- the j,ai\or* market than competencies keyed to 

specific jobs. . •* ' 

In arguing that basic skills attainment is worthy of more attention 

as an evaluation^riterion for vocational education, I should make it clear 

#. 

what I am not arguing fox. I am nojt arguing- for any kind of national standard 
^ of basic skills attainment for vocational education students • ^Vhile federal 
guidelines might, I, think usefully direct more attention to evaluation cri- 
teria in this realm,' I am not suggesting even that the federal government 

55 • 
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make an effort to aggregate results of state and local as'sessjnent of, basic 
skills into a national pi^TCure of the basic skills attainment of vocational 
students. The reasons for not suggesting a ^laxge federal role with respect^ 
to evaluation of basic, skills attainment of vocational stuftents are many, 
but here I mention only three.. First, there seems to be widespread and widely 
accepted agreement to thfe proposition that federal, as opposed $o national, 
standards of educational achievement would be an* unwarranted breach Qjf the 
long-standing division of authority over education in the United States. 
S^econd are the many technical problems that efforts to aggregate state and 
local evaluations of basic skills attainment would raise in trying to produce 
a national" picture of vocational education^ Jhird, J see* little likely 
utility from such an effort in terms of program improvement in light of the 
highly diversified and d ece ntralized nature of the vocat ionajl edur.fitinn eutei^^=-^ 



^rise. . , 

Services' to Special Needs Populations 

A third area for evaluation mentioned- in the 1977 regulations- is ser- 

vices to special needs groups, specifically:^ 

The results of additional^ services, a% measured by the,. suggested 
criteria [pertaining 'to planniitg and, operational procedures, student 
' ^ achievementi,and employment success] that , the state provides 
to these special populations: ' ^ > 

CI) Woman, ^ ^ ' ' " . 

(2) Members of minority groups, ^ ^ 

(3) Handicapped persons, * . 

(4) Disadvantaged persons, and 

(5) Persons of limited English-speaking ability. ' 



*See Jaeger, 1979; Linn, 1979; and Wiley, i979 for a discussion of many of 
these issues in regard to efforts to aggr^ate local and sj>ate evaluations 
of bas^ic skills effects estimates with respect , to Title I programs. 
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*As noted in the review of the Abt ^tudyf and -the Wentling survey in 

section III, evaluation aQtivity regarding services to special populations 

appears to be relatively less than evaluation pertaining either to planning 

arfe operational pi^ocedures, ox to' employment success (though as noted, evidence 

in the Wentling study on the priority given to evaluation of servi'ctes' to 

special populations is somewhat anibigdous). As the Abt r^poxt tioted: *^ 

• Less attention has been given to evalu^ing .the results ^ 

9 . oi additional services to special needs populations • All 

sample states which have a formal program review process • - 

include some items on special needs populations though 

the extensiveness of the review' varies. Ii) all ckses, the 

review focuses on access to rather than results of vocation- 

al programs. (Beuke, 1980, p-*98). ' ' , 

In /this regai'd, it seems to me 'that atll'that' is reasonable'to require ' 

in the way of ^regular evaluation of services to special populations are 

assessments of access to and participation in federally-aided vocational 



e ducat iibnal programs .\ To put the matter another way/ the <1977 regulatory 
language requiring evaluation ofth^^'results of additional se,rvices" 
provided to women, minority groupi^, 'handii:apped and disadvantaged indiv- 
iduals, and persons of limited* Engl^^ speaking abi 1 itj;/4n «mns of employ- 
ment success and student achievement is unrealistic and unreasonable. I. 
reach ^tfiis conclusion for three broad sets qf^reksons. 

^ .Fitst, are th.e general arguments already outlined f ega'rding ^the 1 ignited 
support (either in terms- of financial resources' or tethlnical assistance) with 
whichv the federal government has j^xovided with* respect to evaluation of employment 
success and student achievement effects of vocational education, and the 
limited utility of any^ such evalution in light of the apparent "goal of 

federal evaluation requirements, namely program improvement at state, local^ 
• ' " * ' > • ' ' • • '* 

and institutional levels* A,', . ' « * ' 
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Seco'nd, is the language pertaining tCf^esults of additional services. 
If t^ken seriously, .this wojuld mean that special dotuinen;tation would r>e- 
quired not only of gefieral sjervices provided/to such special populations, 
but df additidnal services provided to each special population above and 
beyond the' basic level of service provision which otherwise might ^e pro- 
vided. This would mean that evaluati^ of the results of additional services 
equire addressing issues of ^^lemental services versus supplanted 
ices - issues which have long bedeviled efforts to evaluate the Title I. 
Third, is the fact that most agencies providing federally-aided voca- 
tional .education programs',' namely local education agencies, already are under 
separate and quite different mandates to evaluate the results of service pro- 
vision to precisely these same special populations, ^F^liave not had space in 
this brief paper to review the various federal mandates for evaluation of 
educational 'service provision to all the special populations mentioned in the 
1977 vocational education regulations, but here let me only discuss the fed- 
eral mandate for evaluation of educjitional progr^s for the handicapped, e^ab- 
orated as a result of the passage of PL-94-142, which was brief ly^ reviewed ^ 
in, section IV.. As noted,-^ federal evaluation 'requirements for LEA' s providing 
services for the handicapped focus exclusively on .prodedural requirements 
(mostly in terms of individual educational' plans) and do not specify atten- 
tion tQ. any particular outcome measur^es. Indeed "Enactment of federal legis-' 
lation regarding 'eyaluation for the handicapped' (PL-94-* 142) s^ems to have 



♦Without frying to Ascribe the* history of the supplement versus supplant 
^iss^e^i'ln Title I, and the various regulations tbat have grown up around - 
. » this, issue, it is worth rioting* that •because of the ramifications bf trying 
to identify supplemental services, the latest" '(1978) legislation provides 
for dropping of the supplemental requirement when Title I services are pro- 
vided to students in districts serving a specified minimum prQportion of 
disadvantaged students. ' ^ 

ERIC . ; ' 58 . « . - 
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\vbeen premised on a potion that any a priori , specification from the federal L 
level of goals for handicapped students would be inappropriate, that the" 
specific criteria for judging tjie effectiveness of individually prescribed 
educational plans for handicapped children should be specified at the loc^I 
level. Thus,, it •^eras to me that requiring th^ evaluation of vocational programs 
serving handicapped children iti.tenns^ of any prespecified criteria of achievement* 
and empk)ynient success would fly in the face of not only the procedural require- 
ments of PL-92-142, but also the general -philosophy which motivated the passage 
of this legislation by the U.S. Congr^s." - ' 

Planning and Opgrationar Processes ^ , v ^ 



The fourth area of evaluation mentioned in the regulatioi^^s of 1977,^^^^ 

was planning and operational procedures. Specifically mentioned_jindfir_thi5 [ 

rubric were the following, sorts of characteristics of vocational eduation 
programs: 

• ' (1) Quality and availability of instructional offerings: 

(2) Guidt^nce* counseling, and placement and follow-up servj.ces; ^ 

(3) Capacity and condition of faparti-^s and equipment; 

(4) . Employer participation- ©Ooperatt-rve pi^^ of vocational 
. " education; J . ' ' 

(5) Teacher/pupil ratios;, and 

(6) /Teacher qualifications • 4' * ' 
Though planning and operational procedures iJere mentioned In the 1977 

regulations prior to the three other areas of evaluation mentioned (namely 
student achievement, .employment success, and results of services to special 
populations), I have postponed discussion of this area of el/aluation for the 
simple reason that I think there is greatest potential in this realm for 
specification of evaluation criteria which will most likely serve the intended 
goal qf federal evaluation , namely program improvement. Why do I 
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give priority 'to evaluation of .planning and operational procedures above^ 
Student achievement, employment success, *^nd results of additional services 
to special populations? Thte reason is" fairly simple. Regardless how good 
estimates of various progjram outcoiies ar,e, they provide absolutely no guid- 
ance for program. improvement unless they are coupled with information on . 
program processes. If one knows that prograip X helped to boost students' 
reading achievement by 10 points, this information is' of little value-if it 
is not accompanied by. information on (^haracteris-tics of the program (i.e. 
operational, processes) which might help explain why' it was effectiye. Iji • 



other words, evaluation evidence on program outcomes such as achievement or, , 



employihent success, however validly manured, provides' virtually" no basis 

for program improvement efforts if it is -not accompanied by information on oper- 



even if one does not have evidence on, outcomes, still can provide a basis for 
improving programs; for example, if some independent standard exists for " * 

judging the characteristics which a program should have or hypotheses are 

/ ' - ^ - ^ \ ' ^ 

available concerning likely relatronsfiips between operational program char- 

Ceteris tics and program outcomes. ' . , / 

In this regard it s^evis to me that 'fe^eral'^.poligy on vocational education' 

( Valuation might pay palrticular attention tq documentation of * the nature and 

extent of the vocational education enterprise. ' Ffom the federal, perspective, 

ijt seems likely to be far more useful to focus evaluation efforts on matters 

vocational education program organizatioiv and administration, populations . 

served and services provided, than on question/ of effects, be they concerned ^ 

with student -achievement, employment success, or results for special populations. 
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There are two jbroad .sets 6f reasons for advocating an evaluation focus on 
documentation of program nature and extent rather than -on outcomes,. This 

first has to do with what has been learned about systematic program evalua-^- 
tion over the last decade and a half and the second with the complexity of. < 
the vocational education enterprise and the federal rc^le in it. - . 

3ince the mid'^lQSOs systematic program eyaluation has emerged as a 
major enterprise. Millions of dollars have been inve^^tecTHn it. Several 
%new* journals expressly intended' to cover evaluation rs^sues haMe been founded 
and evaluation apj>ears to be emerging as a specialized pro5fe^ion, separate 
from research and administration. And as 'noted in sectio/^IT, federal legis- 
lation on social programs now regularly containSTprovisijms cuuceri trng-evalu— 



ation of isuch programs. Nevertheless, not all has gone well' for the ^valua- 



useful, either in terms of, policy-making or program management, as the early 
proi)onents of evaluation suggested '(see Patton, 1978;^ Weiss, 1976, 1977). 

-There are several different ^xplanatipns for why systematic, program evaluation 
has not proven to be as useful as might have been hoped, including, for 
exajnplfe;' the- relatively primitive state of the art of evS.luati<Mi, the failure 
to keep utilization in mind in designing and.^carrying out evaluations, and- 

.l^b myriad other -influences be'sides evaluation findings on policymaking and 
program management .(Lindblom § Cohen,' 1979). Nevertheless, it seems clear 
that one reason why evaluations haye not proven terribly useful is that there 
has been a tendency to focus on questions* of effects^ or outcomes without, 
paying sufficient attention to issues of specific purposes s^d characteristics 
of programs being evaluated, Such, an approach hais come to be known, usually 
derogatively, as' the blackr^hox model of evaluation^ The ma^n problems with 
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such an approach are two. Fi^'St, it does little^ good to know that ptograin X 
produced a gain of .10 points on an achievement t^t, or 10% g^^eater chance 
pf being employed unless such finding's are accompanied -by information on 
* the nature of program X. Second, it is virtually impossible to produce 
effects estimates concerning any regular prograci which cannot be easily 
challenged by anyone with evpn a modicum of knowledge about technical evalu- 
ation issues. The problem is' that without some kind of" experimental manipula- 
tion of assignment of people to pro'grams (for example, via ranclom^ assignment!), 
it is inevitably possible to hypothesize that "estimated effects werei due 
not to .the program under study, but to. the particular characteristics of 

individuals participating in tWe program. ^ For -thfs reason, Boruch and 

• # 



Cdrdray (1980) explp.citly recommended ''the \ise of randomized field experi- 

* ^. » 

ments to plan and evaluate hew programs, new variations of existing programs*, 

and ^program components."- There ^are, I think, strong groionds for questioning 
'this recommendation, ;,but it- is. nevertheless clear tha^t for regular ongoing 
service programs such ^s, vocational education, random assignment cannot (and 
in ray view should not) be employed. This means that evaluations of the out- 
comes prodijced (the effects) of such prog!rams can never be definitively 
assessed.' This suggests to me that evaluation of' vocational education pro- 
grams shoulU focus more on des^criptive questions of what programs are and 
what people they serve rather than- on the much jnore elusive question pf 
, what effects the- programs have.* This is not meant to suggest that questions 

. - ' In this regard ity ^Qxth noting tljat a§ progjauj eyaliaation has developed 
as a separate field of inqiaiiy- several groups haye developed standards for 
progra© evaluation, and prominent aippng theiu are issues of program descrip- 
_ tion. For exanrple, the first standard set out by the Joint Committee on 
Standard?- for Educational Evaluation reads as follows; 

The oBject of the evaluation (program, product cfr project) should • 
be described so that it is clear what formfsj of the object is 
. _ larej .being evaluated.' yoint Committee, 1979) 

. • Similarly, the nu-.iiber one guideline set out in the draft standards developed 
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of impact or effects are not . of interest', hut merely that given the 
last.JLS years of exper^ience with progr^ evaluation^ they bannot^be very 
usefully addressed by mandating evaluation in terms of any one or two 
outcome criteria (nor ' f orn^hat matter any one or two sets of outcome 
criteria).' . ^ ' ^ " ' 

The second brpad set of reasons for focusing evaluation on documentation 
of* the nature and extent of programs rather than on outcomes 1?as tp do with 
the complexity of the vocational education enterprise and the limited federal 
contributioil to that enterprise. As noted in section II, federally aided 
vocational educatiqn serves different kinds of goals (e.g., occupational 
exploitation, preparation for advanced placement, and direct preparation for 
paid or unpaid employment), operates in many different institutions (2^,000 
at last count) and at different levels -of the nation's educational system 
(secondary, postsecondary and adult), concerns dozens of different kinds 
of occupations, and' is specifically mandated to serve seven different special 
populations. In contrast to the scope of the vocatioital education enterprise, 
however, federal aid to vocational education amounts to no more than* 10% 
of total national p;xpenditure on vocational education. In light of this 
contrast, and in lights of the fact that the federal government has not , ' 
provided significant support -tp^ states, local education agencies and . 



*(cont.l*: i - - ^ 

by the evaluation research $oc?:ety is; 

V The purposes and characterj;$tics o% the program or 

actilrity to be addressed^in the evaluation^ef fort should 
^ -"bV specified as precisely a$ possible (.cited in Boruch and 
Cortlray, 1380, appendix)^, — 



institutions to meet the evaluation- mandate in 'the 1976 legislation, much 
less the elabcfrated mandate in the 1977. regulation, it se^^slmreasonable ^ 
for federal Ifegislation to r-equire evaluation in terms of any specific 
criteria or set of criteria. - • ' n - 

ivhat Else. Can be Done to Make "Evaluation Useful for Program Improvement? 

Thus f3r I have argued mainly for curtailment of .federal requirements 
for evaluation o^ vocational, education outcomes. Nevertjieless, thfe question 
of results is clearly of tremendous importance. The national interest ob-- 

% , 1 ■ ' ' 

viously will not be well 'served by' providing' vocational education programs 
if those programs make no difference in the lives of individuals whom they 
aim at serving. Thus, in closing, it' seems to be worthwhile to move^eyond 
the fairly narrow question of evaluation criteria, and more broadly address 
w iaicA <»oT» Vin rinnn 1-n maVp pva 1 Hat T '^T! useful for uroGfram 

improvements. WhilQ. I do not haye time no^ space to. elaborate much on this 

question, let me suggest five broad ^types of possibilities. 

' " . ■ \ 

E valuajltjLon Procedures > \ One strategy, apparent in- the *L-94-142 * 

legislation, would be to eschew the mandate of evaluation in terras of any 
specific criteria, a^d instead to place more ejnphasis -on procedural require- 
ments. Who should be' involved in the coAduct and reporting of evaluation 7% 
and who should have access to evaluation Vesults^ Instead of viewing former 
vocational education students (or in the jargon of jcurrent literature, 
program completers and leavers) <^y.,as obljects^of §tudy, might evaluation 
policy seek to ascertain their vie^s more directly on. the strength arid 
weaknesses of the.'prograjns in wMxh they pelrticipated? Might not parents 



of students, employers, and representatives of special populations, all 

• - • »" 

of whose interests are intended to be served hy federally aided voca^r 

tional education, be sy.stematicaljy involyed in d^aluating this complex 



enterpriser ' ^ ^ 

It should be noted that some such procedural requirements for evalu- 
ation already exist • * As (noted in the general description of federal 
requirements in appendix 1, for example. State Advisory Councils on 
Vocational Education ^already are mandated to participate in evaluations^ 
of vocational education "^at state levels', and the National Advisory Council 
on Vocational Education serves as the national counterpart to the /state 
councils. Also, as noted in the Abt report (Beuke et al,, 1980, p. "^1,08) , 
the Education Amendments of 1976 ^and 1978 mandate the establishment- of . 
three other sorts of' groups at the state^JfeVel {Committees for State 
Vocational Educatioi^al Planning, State Occupatioir^al Information Coordinating 
Committees, ^nd Statet Employment and Tr^aiiii^g^ Councils) in ||^der to co- 
ordinate CETA and vocational education programs,^ The establishment- and 
operation of such coordinating groups has not been without problems, as 
the Abt report notes • ^ But theii: existence (and according to' 
the Abt report they ''are having positive effects"^ Beuke et al., 

1980, p. 123), clearly suggests that the locus of ^responsibility for 

' ' ' ' "* ' ^ 

.vocational'* educational plainning and policy-making resides 'more at the state 

than the federal'^ level of goyea^ment. If this is so, it would seem plifusibl 

to avoid direct federal mandate for evaluation in terms of any specific. 

criteria, and instead to rely jnore on procedural guidance from the federal 

level/ ' ' ' 
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Eaemprtgry Program Identification, A second strategy would be to 



provide for Incentives, Beyond rules and regulations pertaining to 
specific evaluation criteria, to motivate useful evaluations of vock- 
tional education programs. One example of such a strategy is available 
in the Department of Education's Join^t Dissemination ^Review Panel (JDRP). 
The JDRP regularly reviews evaluation evidence on the effectiveness of 
.education programs in' order to judge that evidence's persuasive. If a ^ - 
program is approved by the JDRP, it is listed in a Department o^ Education 
catalog. Education Programs That Work , and is el^ible for federal suppprt 

• to promote aissemination of information about the program* The fifth * 
edition of 1 Education Programs That Work (1978) lists' over 2G0 "exemplary * 
educational programs,*' but among these o#ly a single one is listed in 

. the index Under vocational education.*' In contrast, exemplary programs 
listed under the rubrics of basic skills and early childhood education 
number more than two dozen each. There are several <iifferent Hypotheses 
available to explain why vocational education programs ^how up rel-ativ^ly 
infrequently in this ^listing ^of exempl^ educational programs,** but it 

* It IS the Vocational ^Reading Power project developed in Minneapolis*, * 
^Minnesipta, a state with .one of the best developed systems for evaluating 

, vocational education programs. 

**Such hypothejses include: the apparent* isolation iji. the federal bureau-^ 
cracy of vocational education from other education programs, the relative 
prominence of state as /opposed to federal authority in vocational educa- 
tion, the emphasis of "the JDRP on^^educational significance'' as opposed 
. to broader social and occupational significance, and greater federal 

• investment in e^faluating educational programs other than vocationaJ 
programs. 
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^® '^^^y imlj;kelj^ that it is hec^u^ie vocational education'^ prograrosv 
are genericali/ le^s effective or exmpldry than other educational programs 
|ut whatever the cause, ^this fact suggests that vocational education has 
not exploited evaluation as a inieans of identifying good programs, publi- 
cizing informatipn aSout them and thereby improving^ other programs, to 
^s great an extent as have other feiderally aided education programs. 

Technical Assistance . A third approach to evaluation which seems 
to roe mor^ likely beneficial in terms of helping to inqjrove programs at 

state and local levels would be federal provision of technical assistance 

< ■ s 

with respect to evaluation. To some' extent this, is of course already 
happening through activities/>-of the National Center for Research in 
yocajidnal Education. This Center currently makes availat)le a* variety 
bf roatjerials concerning,j^6cational education evaluation (for example', 
concerning the roeasureroent of vocational education outcomes, and^the 
characteristics of programs which tend to have" high placement rates). 
Nevertheless,^ the federal activi"ty in proy,iding technical assistance 
with regard to Title^I evaluations, through ten regional technical ^ 

» 4. <» .V. 

assistance 'centeK, provi'des an interesting contrast. The indde'l Is of - 
course not necessarily one to be directly emulated with respect to 
vocational' education; but it ii, I think, at least worth considering 
in an*effbrt to identify^^ayV in which federal provision of technical 
assistance With Regard to evaluation xan help states and local iristitu- 

\' - : - ' ^ - . ■ - 

tions use evaiuation for the purpose of improving programs, ^ 
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Evaluation Research into .Specgal Issues ^ A fourth strategy for ^ 
• '' / - ^ , ^ 

federal influence on. evaluation of vocational education is .targeted 

' ' . . - ^ • ' ' 

evaluation research on special xssue§, - In this regard, the^very com^ 
pl^xity of the vocational education enterprise represents the various 
and soinetiines competing interests involved in vocational education ~ 
interests of federal, state, and local levels of government, interests 
of individual students and^'parents and those of diverse institutions,* 
interests -of bysiij^ss and organized .labor, and various spWiar interest 
^groups'as well. Current %deral requ^ements for the Waluatibn^of 
vocational education, i»t seems to ine, clearly represent many of these 
different -interestk*, and vihiie Ivhave argued against^ federally required 
evaluation criteria representing various, special interests the 
results of vocational adu|||^on services provided to special populations), 

I am not arguing against th^ va lidity o f the i n>te rests they ^ represent, . 

Rather T am simply^ arming that it is needlessly burdensome, (and- I suspect 
highly inefficient as well in terms of deriving valid answers to specific 



questions) to require all institutions receiving federal vocational educa- 
tion funds* to^gularly evaluate their efforts in terms of such diverse 
Criteria a^ currently mandated. ' Thus it seems to me that a more reason- 
ablQ abroach to^eriving answers to specific quest^ns (such as results, for 
pa^^cular special populations) would be targeted evaluation rfesearch^ aimed 
at answering specific questions. Such a strategy could be easily cbprdi-r 
nated with a federal approach to evaluation of vocational educ^tion^ 
which focuses on documentation ai^jj^descriptipn of vocational education 
administration, services and populations. If such descriptive information 
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WQre available, it could be used to provide a framework in terms of 
which evaluation research targeted on specific questions could be organ- 
ized, both in terms of elaborating specific questions and in terms of 
drawing systematic samples of vocational education programs and popula- 
tions. 

In suggesting the strategy of targeted evaluation research, I should again 
point out that the idea is not a new one. Indeed, the current vocational 
education sfudy poses a prime example of such targeted research. Instead 
of relying exclusively upon the ongoing evaluation activities of the 
federal government', as in thg Education Amendments of 197,6, the U.S. 
Congress specifically mandated (and funded) this Study to address specific 
questions through "thorough evaluation and study" whici/ the Congress wanted - 
answered in preparation for reauthorization of fe^ier^ legislation on 
vocational .education. Also, the data now being acquired through VEDS • 
provides an example of how descriptive data, acquired at the federal level, 
"canTielp provide a framework for and thereby help refine evaluation 
questions addressed at the state and" local levels. In this regard note 
that while NCES in reporting in 1980 on The Condition of Vocational Educa- 
tion drew a broad distinction between occupational and nonoccupational 
vocational education programs, officials from the state of Minnesota re-« 
ported data showing that the amount ^ coursework students take 'in voca- 
tional education, in Minnesota spans a\pntinuu^ranging from l4ss than 



100 hoars to more than 80a -hours- C^ee u\. Congress, p, 38*4}. In other 



words, while the general distinction drawn^t the federal level is a useful 



one Cfar better than lumping together all secondary school students 
taking vocational coursework as simply 'Vocational education students"), 
a more detailed description developed at the state? level revealed vthat 
such a dichotbmization 'still conceals considerable variation in the 
extent of the vocational education training received by students. 

Meeting Labor Market Demands , A last general issue pertains to- the 
widespread interest in using evaluation to help improve the coordination 
of vocational education with labor market demands. As noted in section I, 

A 

the widespread interest in making vocational "education program offerings 
responsive to changing demands of the labor market is entirely under- 
standable^ It does little good to prepare individuals for occltpations 
where there are no jobs available, . Also, as noted, the concern for 

making vocational education more responsive to the labor market was 

clearly one of the prime motivations in the 1976 Congressional mandate 
to evaluate occupational vocational programs in terms of stucients^ sub- 
sequent employment success (namely, placement in a related occupation, 
and employer satisfaction). Nevertheless, I also suggest that follow-up 
studies 6n employment sucfcess are not a terribly satisfactory means of 
serving the intended end -- both because of practical problems inJ 
conducting such follow-up studies while protecting the privacy rights of 
individuals, and more importantly, because such follow-^iq) studijes on 
former vocational eclucation students inevitably will miss' half the 
question. Specifically te'lying on foUow-eup studies will provide no 
'direct information on occupatiortal openings . fori which there ar.e currently 
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no vocational program offerings. For example, in community Z, there 

may be a tremendous demand for computer programmers, but if. thef e is no 

vocational program related to con5)uteT progr'araraing in that community, 

this fact would very likely not be revealed in follow-up studies. 

If this is so, there remains the broad question of how mechanisms 

can be established to help make voca^tional education program offjgrings 

more responsive to labor mv]cet opportunities. The general answer, I 

think, is to bring other kinds of data to bear on vocational ^education 

planning and policy-making, namely surveys of current labor market 

opportunities and projections of future labor market demands. This 

idea is of course' not a new one. It was reflected, for example, in the 

1976 Congressional mandate to establish State Occupational Information 

Coordinating Committees, with membership to include state administrators 

.of vocational education. Nevertheless, it should be acknowledged that 

the idea is not easy to .implement for at least two broad sets of reasons. 

First, from a theoretical point, of view,' there is considerable uncertainty 

concerning market elasticities of skill substitution within or across 

occupations. As Thurow (1979) points out,^. ' , 

The empirical magnitude" of the elasticities of substitution 
depends on the level of aggregation of labor'skills* At 
some levels of aggregation the elasticities are low; at 
others they are high. Generally, the more aggregate the 
vector of labor skills, the lower the elasticities of sub- 
stitution between different skills* The elasticities of 
substitutiort between different types of electricians are , 
presumably much higher than those between craftsmen and 
professional' workers, Cppt 323-^24} 
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in general Thurow seems to ar^ue that elasticities of skill substitution 
are higher thayT^ypically recognized in vocational education and enqjloyment 
'training circles. Nevertheless, at the same time he acknowledges that for 
some categories of skills, such as typing, elasticities of substitution 
may be low, ' * , ' " ^ 

Yet leaving aside such theoretica.1 problems, there is a more practical 
prob^edl in.attainjlng the goal of making vocational education more responsive 
to labor market demands* 'it is the fact that, whatever one wishes to assume 
about elasticities of skill substitution, there is a tremendous dearth of 
information systematically collectied on labor market demands. Indeed, 
according to some, recent accounts, the United States has the poorest 
system of collecting such information of any major industrialized country,* 

This suggests that meeting the goal of making vocational Wucation 

• «? . , 

more responsive to labor market, dcfmands would require the investment of 

/ . _ . ■ • ' 

considerable resourced in developing a more comprehensive system for 
collecting, analyzing and reporting data on labor market demands. Such a 
task clearly seems beyond the ken of vocational "education, or at least 
vocational education alone. Nevertheless, it seems to me that, however 
such a goal (namely'^e development of a better system for ascertaining 
present and future laltor. market demands) can be attained, at least one 
thing is clear. Liabor mairjcet demand, as with other aspects of the economy, 



can fluctuate markedly across time and geography; According to the 



i 

Department of Labor (J980}; ^ ' 



In every occupation and industry, the number of 
jobseekers and the number of job openings con-^ 
stantly changes, A rise or fall in the demand for 
a product or service affects the number .of workers 
needed* to produce it. New inventions and tech- 
nological innovations create, some jobs and eliminate . 
othersl' Changes* in the size 'or age distribution of 
the peculation, work attitudes, training" opportunities, 
. \ or retirement programs determine the number of workers • : ' 
available. As these f6rces. interact in the labor ♦ ^ 
market, some occi^ations- experience a shortage of 
workers, some a surplus, some a balance between job-. , 
seekers and job openings. Methods used by economists*' 
to develop information on future occupational prospects ' 
differ, and judgments that go^into any assessment .of 
,the future also differ • 

This suggests" that the task of coordinating vocational education 
programs offerings witli current, much less future, labor market demand 
is not an easy endeavor. It also suggests to me that mandating indi-^ 
vidxial education agencies and institutions to tailor their vocatiohal 
education programs to labor market trends by way of their own folldw-up 
surveys of former students is simply ' unreasonable: .Instead I thiiik a 
more reasonable jstrategy would be to invest additional resources in state 
or regional efforts to gather, analyze and report on labor, market trends 
and 'to influence vocational education planning efforts by providing 
such- information to local institutions. 
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Evaluation Requirements for Vocational Education 

Funding for Federal Vocational Education programs Is of two basic 
types: Formula grants to states and Discretionary grants. The evaluation 
process Is different for each type. Here we only consider, the evaluation^ 
requirements f(5r the formula grants administered by the at'at^s^ 

State adplnlstered Vocational Education programs require evaluation 
at the state and federal levels. At, the state level, formal evaluation is 
routinely conducted by two groups; the State Department of Vocational 
Education and the State Advisory Council on Vocational Education (SACVE). 
At the federal level, there is a parallel organizational scheme. The Bureau 
of Occupational and Adult. Education (BOAE) within USOE and the National 
Advisory Council on Vocational Education (NACVE) serve as the federal level 
counterparts to the state agencies. The local administration of these 
programs is carried out by the district. The evaluation is typically 
Informal, being composed of needs assessment apd guidance regarding program 
operation provided by tHe Local Advisory Coun/ll on-Vocational Education 
(LACVE) . • , _ ■ 

i , ■ »• 

• Evaluatio n Requirements at the State level . The 'law and. regu- 

lations are explicit as to the content and procedures to be employea in 
the state evaluation. The evaluation is 'structured around a five-year 
program plan. The legiajation explicitly states that the purpose' of the 
evaluation is to revise and Improve the programs conducted under this -plan, 
this plan is jointly devised by representatives of the State Department " 
of Education and the State Advlsor^Councll (SACVE). 

Stqte De partment of Education requirements . During the five-year ' 



period of the state plan, the State Department of Education* is to evaluate 
the effectiveness of each program in terms of (a) planning and bperatlonal 
processes, (b) student achievement, (c) student emplpynent success, and 
(d) issues related to sieclal populations. Further,' the state is required 
to evaluate the extent to whlcWMndlvlduals who complete or leave the progyam 
obtain employment in occupations related to their training and whether their 
employers consider them well-trained and prepared for emproyrtent . ■ Sampllrfc 
is permitted for this assessment. Finally, the State Department of Education 
is required to submit, an annual accountability report which Includes a 
descriptioo of how funds were used, a summary of the evaluations that were ' 
conducted and a description of how the evaluation Information, has been used 
to Improve the state's program. 

State Advisory Cou ncil requlrements « Annually, the State Advisory 
Council is to prepare and submit to the ' Commissioner and National Advisory 
Council on Vocational Education, an evaluation report. Its contents are to 
Include a synthesis of its evaluation of State Department administration and 
operation and the evaluations performed by the State Department of Education. 

'^^^ Evaluation Requireme nts at the Federal level . An organisational 
structure.' parallel, to the state level, is. established within the law for the 
federal. level agencies. There are" some notable dlffernences in the expllclt- 
ness.of the- evaluation-requirements, prescribed for 'the National Advisory 
Council , however . • ■ , 



* Evaluation requirements for the Bureau of Oecupatironal and Adult 
EdjUcation * At least ten states are to be reviewed dviring^a given fiscal' 
year. The purpose of the review is to analyze the strength and weaknesses 
of state programs. At tbe same time, DHEW is to conduct fiscal audits 
within those states. The Commissioner is to transmit to Congress a report 
on the National status -of the Vocational Eclucation programs. The report is 
to include information developed from the National Vocational Education 
Data System (VEDS), a summary of information obtained from federal reviews 
and audits and a synthesis of the evaluations performed by State Departments 
,\nd State Advisory Councils. . * 

Evaluation requirements for the National Advisory Council on Vocational 
Education ^ NACVE received a brbadly stated evaluation function in the legis- 
,lation. Its primary function is to provide policy-oriented annual reports 
and assessment pf USOE/BOAE administration and (jperations* 

Diversity in the Type of Evaluation Regulations 

Examining the amount arid type o£ iTiformation that is required across 
the four programs it is apparent that .there are .substantial differences* 
The direct grant type of program (e.g,, Bilingual and the Discretionary 
grants for Vocational Education) have the least amount of oversight and ^ 
^reporting requirements. Title I and Vocational Education (Basic grants) 
are both state administerecl , formula allocation grants and have an additional 
level of evaluation imposed. by the state agency. Vocational Education can/* 
be distinguished from Title T in -that' two Agencies at the state and tWo \ 
agencies at the federal level are responsible for conducting routine evalu- 
ations • From this comparative assessment^ -we see that hot only do the law 
and regulations indicate .how evaluation is to be carried out, it can also 
influence how mj^ch is conducted and by whom. 
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' ■ ESEA,. Title I (Educ ation of DlsadvantaRea Children, Basic- Gran ts 'to 
LEAs ) '• : ■ 

-LEA Evalu ation Requirements . The 1974 and 1978 Educational 
AoenamentS require the Commissioner to develop and make available to SEAs 
and tEAs (through th» SEA) explicit standards and "models" for evaluation 
reporting at the local level. The October 12, 1979 Federal Register 
describes these standards and reporting regulations: . eyery LEA receiving 
funding is required to submit an evaluation plan to the SEA that addresses 
how it will meet technical requirements of the regulation. "At least o^^ 
every three years, the..ljEA,^must evaluate its programs using, "reliablt 
"m*?^*^ ^"®^'^"°^"^^»" "procedures that minimize error" and a de sign t hat 
yields a valid assessment of achivement gains." "fhis latter requirement 
can be fulfilled by using*»pne of three federally developed models or a 
suitable alternative approveTd by the SEA and Commissioner. Each model Is 
supposed to'provide an estimate of the effect of receiving Title I services 
on student performance compared to an estimate of what performance would 
have been in the absence of Title I services. Achievement scores are to 
be reported to the" SEA using a common meascKig, a "normal curve equivalent" 
(NCE). ' • ^ . . 

-=>The new regulations also require longitudinal assessment to ascertain 
whether Title I gains are sustained after services are .withdrawn. ' This 
ass'essment is for -local use and reporting is not required unless requested 
by the SEA. Initial achievement status and gainj a description of the 
assessment process and project infotinatipn are the only federally mandated 
.evaluation requirements -that are imposed on LEAs. The project information 
that is to be obtained includes: average duration of Title I service, , 
pupil-per-te^che*^ratlos, expenditures per child, and number of participants. 
According. to .the^egftl^tions , this projett infonnation is to be collected on 
sample of grade level's*' *■ 

(^) The SEA Evaluati6n\tgquire^)Bents . The SEA is charged with the - 
responsibility for ensuring that the LEA educational plan is in compliance 
with, the-law-and recently, this rolfe has been expanded to Include more 
extensive evaluation functions. SEAs are responsible for monitoring how 
the projects are carried out, providing technical assistance regarding LEA 
evaluation and aggregation of. LEA data. The monitoring function is carried ' 
out through field visits by state Title I representative (s) . The state 
receives pne and one-half percent (set-aside) 'of its total allocation, or 
5150,000, whichever is greater, to perform these^nctions. 



The . SEA .-compiles the data that is submitted by the LEAs "and submits 
annual performance report, containing: The number of participants 
serviR'd by types of service; number of .participants by grade level for public 
and nonpublic recipients and "otjher Information requested by the Commissioner 
and (2) ^ biennial e valuation report , summarizing Information for all or a 
representative sample of LEAs. 



(c) Federal Evaluatlcj^ Requirements . Sedtlon 183 of the 1978 
Education' Amendments clea^y delineates the evaluation tasks and priorities 
to be addressed by the Commissioner- The law makes provision for two levels 
of evaluative evidence: independent evaluations designed to "describp and 
measure the impact of ptdgrams" and the Provision of Technical* Assistance 
to States and local agencies on conducting evaluations. A maximum of one*- 
half of 1 percent of the amount appropriated for these programs Is provided 
for evaluation, and, priority Is to be given to the federal assistance to 
state and local agencies. 



APPENDIX 3 



Public Lav 94-142 (Education for the Handicapped) Evaluation 
Requirements ^ ' ~^ ^ 

The Education for all Handicapped Children Act of 1975 and the per- 
tinent regulations' are explicit about responsibilities. The states are-, 
the primary target of federal oversight arrd thev in turn are responsible 
for overseeing the/local education agencies. Th^ program Is focused on 
the provision of/a ''free, appropriate public education for all handicapped 
" chll.drepr:? The Bureau of Education for the Handicapped (BEH) InXUSOE was 
assigns the responsibility for admlnlstratlbn and evaluation of K.L. 94-142. 

(a) LEA "Evaluation** Requirements , At the local level, the term 
-evaluation refers primarily to diagnostic assessmeot of children. The 

regulations require that preplaceiuent evaluation be conducted using 
multiple, appropriate assessment modes. If the child is found to have 
a handicapping condition, an Individualized Educational Plan (lEP) .is 
devised. The content of the Individual Education Plan is required by the 
regulations to include: (1) an assessment of present levels* of educational 
performance; (2) a statement of annual goals and .short term Instructional 
objectives; (3) a statement of specific special education and relatted services 
and an assessment of the extent to which the child is able to participate in 
regular education programs; (4) projected dates for initiation and termi- 
nation of services, (5), appropriate objective criteria, evaluation proced- 
ures and a schedule for reevaluation. ^ 

(b) SEA Evaluation, requirements , fhe state has responsibility to 
ensure that the lEP^ has been prepared and that it meets the educational 
standards .of the state. This is essentially a monitoring function and is 
carried 9ut through on-site visits. Elaborate checklists haVe been developed 
by state agencies- and BEH for assessing compliance with regulations. Addi- 
tional monitoring requirements include fiscal audits and an assessment of 

the extent to which the Individual Educational Plan is actually . carried 
out, in practicjB. This rlatter function is essentially a check tS ensure 
that the p^gram for individual children is actually implemented. 

The law specifics that in any fiscal year, the State may use five* y 
percent of the total state' allotment under part B, or $200,000, whichever^ 
is greater for conducting required administrative activities . Evaluation 
In the sense of monitoring is included under this category* of activities. 

The State Education Agency .is required to report (1) the number- of 
handicapped children receiving services on October 1 and February 1 of the 
school year; (2) the number of handicapped children within each disability 
xategoty; (3) the number of handicapped with each of three age groups. 
For all figures, undupllcated counts are required. This report is to be 
transmitted to the Commissioner. ' * " 



Evaluation requirements at the Federal tevel . The Cotnnissioner 
has re«pon«lbility for evaluation under Section 618 of the Act. Specific 
cally, the legislation authorizes (1) annual studies; (2) assessment of 
the adequacy of information provided by state agencies; and (3) development 
of effective methods and procedures for evaluation. 

This degree of explicitness is oft^ not warranted for a variety of 
reasons. In some cases, the feasibility of conducting a^pecific type of 
evaluation is unknovm an^ the details are deferred to th^ agency responsible 
for carrying out the evaluation. In other instances (e.g., ESEA, Title I) 
the law requires the development and implementation of models for estimating 
program effects but the SEA and LEA personnel are provided with some dis- 
cretion* as to which model they will follow. 
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